Hello everyone, This mail is kind of late, due to the recent Easter holidays, sorry about that. New DebOps release ------------------ About a week ago I released a new version of DebOps, v0.7.1 and immediately v0.7.2 to fix an issue in the Docker image build. This slip up was due to how the code is currently tested on Travis, and the test suite will need to be overhauled during the next release cycle to avoid similar issues in the future. You can find the new version of DebOps on: GitHub: https://github.com/debops/debops/releases/tag/v0.7.2 PyPI: https://pypi.python.org/pypi/debops/0.7.2 You can also install the new scripts by running the command: pip install --upgrade debops You can find the brief Changelog in the DebOps documentation page: https://docs.debops.org/en/latest/news/changelog.html Complete, detailed changelog can be viewed using the 'git log' command. The DebOps documentation has a separate page which details important changes from previous release in the Ansible inventory or on the remote hosts which you might need to perform manually: https://docs.debops.org/en/latest/news/upgrades.html The Python packages available on PyPI, as well as the tarballs available on GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver network using the command: gpg --keyserver hkp://pool.sks-keyservers.net \ --recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC The new release has a relatively small number of commits since the previous one, 167. List of commit authors without merges: 122 Maciej Delmanowski 15 Robin Schneider 6 Alin Alexandru 4 Jérémy Gardais 3 Stefan G. Weichinger 2 Gaudenz Steinlin 2 Jan Katins 2 Jason Goldfine-Middleton 2 Peter Niederlag 1 Joseph Rawson 1 Konstantinos Koukopoulos 1 Markus Wigge 1 Oussema Cherni 1 Pedro Luis López Sánchez 1 Russell T. Sherman 1 Scott Saunders 1 Serge Victor 1 Werner M. Krauß Big thanks to everyone who participated in this release! Code linting and cleanup ------------------------ Most of the changes were related to linting existing code by various tools. Right now, yamllint, pycodestyle and shellcheck are active during Travis CI tests and any errors detected by these tools result in a failed test. To avoid that, I suggest that you check the code beforehand locally before commiting. There are sill some '/bin/sh' scripts which were omitted due to insufficient detection, they will be fixed during the next release cycle. This release comes close after the Ansible 2.5 stable release, which is now the default target for DebOps. Due to that I cleaned up most of the deprecation warnings in various roles, which should help with finding new ones in the future. The missing inventory groups still emit warnings during execution, however that has to be dealt with on Ansible side. Role directories were cleaned up and the unnecessary '.travis.yml', 'README', etc. files were removed. I'm still not sure how Ansible Galaxy team plans to design the support for multiple roles in a single git repository, therefore they might come back later. Installation improvements ------------------------- The 'debops' script can now correctly point Ansible to roles and playbooks included with the Python package installed from PyPI. This allows for simple stable releases which you can contain in their specific Python virtual environments. You can still download the latest DebOps monorepo using the 'debops-update' command, which will take precedence over the packaged version. DebOps now depends on the 'python-ldap' library to allow correct usage of the 'ldap_attr' and 'ldap_entry' Ansible modules. Unfortunately this changes the installation procedure a bit since 'python-ldap' Python package does not provide a "wheel" (binary package) and needs to be compiled from source, therefore external dependencies are also required. The exact details will be explained in the installation instructions, for now installing 'python-ldap' APT package should be enough if you install DebOps from PyPI system-wide. User or virtualenv installation requires a development environment. DebOps should be now mostly compatible with Python 3 environment. If you find any issues while using it with Python 3, let me know. Currently various DebOps roles install Python 2 packages by default, however I plan to introduce a way for the roles to selectively install Python 2 or Python 3 packages. I hope that this allows Py3-only Debian installs in the future. New Ansible roles ----------------- Some of the 'debops-contrib' roles were finally integrated with the main DebOps playbooks. The 'debops.etckeeper' role was slightly redesigned and integrated with Ansible local facts to track any changes between Ansible runs - the modifications will be committed automatically when Ansible gathers host facts. Etckeeper is now enabled by default on all DebOps hosts, but that can be disabled if needed. The 'kernel_module' contrib role was redesigned from scratch and is now included as 'debops.kmod'. It can be used by other roles to manage kernel module configuration and load modules on demand if the host supports it. The 'debops.ansible' role is back, and can now install Ansible in multiple ways - either from Debian Archive via the backports repository, from the Ansible upstream APT repository, or via a locally built '.deb' package. The new 'debops.apt_mark' role can be used to change the APT package installation status (automatic, manual) which is useful if your fresh Debian/Ubuntu installs end up with many packages set for automatic removal. The role is included in the common playbook by default, however there are no specific APT packages marked, you should prepare a list yourself in the Ansible inventory. Changes in existing roles ------------------------- A significant change in the PKI environment managed by DebOps is addition of wildcard host subdomains. Previously, the generated X.509 certificates had only '*.example.org' wildcard which allowed configuration of services for the entire domain. The new X.509 certificates, in addition to that, will also have the '*.<host>.example.org' wildcards which should cover services specific to a particular host in the cluster. This should allow easy integration of per-host monitoring services like Prometheus, Cockpit and the like. If you want to enable the new functionality in existing PKI environments, recreating the "end-entity" certificates should be enough, there's no need to recreate the entire CA infrastructure. ACME v2 is now supported via the 'acme-tiny' Python script - existing ACME PKI realms will need to be recreated to support new Let's Encrypt API endpoints, see the upgrade notes for details. There were significant changes related to the '/proc' filesystem in the 'debops.proc_hidepid' role. DebOps now configures the 'procadmins' group with a static GID 70, which allows synchronization of the '/proc' access between a LXC host and LXC containers. The role will also ensure that the /proc/sched_debug' file is not world-readable anymore to disabllow information leakage to unprivileged users or unprivileged LXC containers. The 'debops.proc_hidepid' role will also check if '/proc' filesystem changes are allowed, which should improve support for DebOps in unprivileged LXC containers. The bug in 'debops.nginx' related to change in Jinja2 library behaviour has been fixed, you can now use Jinja2 2.9+ versions without issues. The 'debops.sysctl' role was refreshed and now uses custom filter plugins provided by DebOps to manage configuration. It will also check which kernel parameters can be modified in the current context (host, VM, LXC container) and only configure the parameters that are allowed. A custom hook system has been added to the 'debops.ifupdown' role. The first hook, 'filter-dhcp-options', can be used to selectively apply DHCP options received from the DHCP server per network interface - useful with multiple networks with DHCP servers. Plans for the next release -------------------------- The test suite used on Travis-CI needs to be revamped because the test installs an old version of the 'debops' package instead of the "current" one. Due to this the above mentioned quick fix was needed. Additionally, at the moment the whole test is repeated three times, however that doesn't have to be the case - linting, Docker image and documentation could be tested just once, in paralell with different Python version tests. Since Python, shell and YAML code is covered in the test suite, I would like to add 'ansible-lint' with a set of custom rules that describe the current DebOps practices. Additionally, roles could be verified with 'testinfra' and additional set of tests after the playbook execution is done in GitLab CI. There's still lots of documentation to be written, especially that the installation instructions need to be updated. In this release cycle I'll try to move the DebOps Guidelines and Policies into their proper place in the docs, so that they can be properly referenced. Some of them will most likely receive their own 'ansible-lint' tests as well. Recent changes at work prompted me to rethink now DebOps manages networks and IP forwarding. Right now in the 'debops.ifupdown', 'debops.ferm' and some other roles there exist a concept of "public" ie. default, and "private" ie. additional network, but this is counter-productive in more complicated environments where, for example, the default network is internal and therefore not "public". I'll try to revisit these roles and see if they can be changed to manage networking more effectively with the new concepts in mind. IP forwarding is currently enabled globally in 'debops.ferm' role, however this seems to be a bad fit since the role needs additional variables to track the state of forwarding between runs. Moving the forward configuration entirely into 'debops.ifupdown' role and making it configurable per-interface seems to be a better solution than the current one. The improved 'debops.sysctl' role can then be used to configure IP forwarding in the kernel per interface, via generated dependent configuration similarly how 'debops.ifupdown' is currently configuring 'iptables' via generated configuration for 'debops.ferm'. See you on the next release, Maciej