[debops-users] New DebOps stable release - v1.2.0

Hello everyone, The next DebOps release is here! I kind of forgot that December 1st is on Sunday and not November 31th, but I hope it's close enough. New DebOps release, v1.2.0 -------------------------- You can find the new version of DebOps on: GitHub: https://github.com/debops/debops/releases/tag/v1.2.0 PyPI: https://pypi.python.org/pypi/debops/1.2.0 Galaxy: https://galaxy.ansible.com/debops/debops/ (but see below) You can upgrade the Python-based installation by running the command: pip install --upgrade debops The support for Galaxy Collections has been improved, but there are still issues - namely, Galaxy does not support role dependencies properly[1], and because of that the 'namespace.project.role' role naming scheme cannot be used in the playbooks yet. Installation via PyPI package or directly from GitHub repository should be preferred this time around. Installation instructions can be found here: https://docs.debops.org/en/stable-1.2/user-guide/install.html The brief Changelog can also be found on the documentation page: https://docs.debops.org/en/stable-1.2/news/changelog.html Complete, detailed changelog can be viewed using the 'git log' command. You can use the 'git log --no-merges' command to skip the "boring" merge commits. The DebOps documentation has a separate page which details important changes from previous release in the Ansible inventory or on the remote hosts which you might need to perform manually: https://docs.debops.org/en/stable-1.2/news/upgrades.html The Python packages available on PyPI, as well as the tarballs available on GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver network using the command: gpg --keyserver hkp://pool.sks-keyservers.net \ --recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC Since the v1.1.0 release in August 2019, there were ~504 commits in the DebOps repository, not counting merge commits. Here's the breakdown of the committers in the v1.2.0 release: 306 Maciej Delmanowski 108 Rainer 'rei' Schuth 21 Robin Schneider 15 Tasos Alvas 13 Imre Jonk 13 Nicolas Quiniou-Briand 8 Leonardo Bechea 3 Douglas Heriot 3 Hartmut Goebel 3 Thomas Finstad Larsen 2 Aljosha Papsch 2 Gaudenz Steinlin 1 Dionysis Grigoropoulos 1 Hendrik Visage 1 Hüseyin Uslu 1 Patryk Ściborek 1 Rolf Morgenstern 1 Rostislav Kandilarov 1 Thomas Danielsson Thanks to everyone involved for helping shape up this project, and see you in the commits. [1]: https://github.com/ansible/galaxy/issues/1938 Goals reached since previous release ------------------------------------ As usual, only some of the previous release goals[2] managed to get done. The redesigned 'debops.golang' role is one of them, and new roles for MinIO and MinIO Client (mcli) use it to install the upstream applications. More Go-based applications will follow in the future. The roles for OpenNebula didn't make it this time, and with the current plans I'll probably leave this for after v2.0.0 release in January. The same goes for improved Samba support - unfortunately I didn't get to updating the 'debops.samba' role yet, or even integrating it with LDAP. It will have to wait. There's also no .deb package support at the moment, perhaps in the next release. [2]: https://lists.debops.org/pipermail/debops-users/2019-August/000243.html Reshaping the project around LDAP --------------------------------- During this development cycle, lots of work happened in regards to the LDAP directory support. The initial directoory layout was redesigned around Groups and Roles, Access Control List was revamped a bit to support hiding LDAP objects, 6 DebOps roles have been either converted to use the new 'debops.ldap' infrastructure, or LDAP support has been added to them... There were many changes in the 'debops.slapd' role as well, from new LDAP schema installed by default, to support for ACL rule tests. If you use LDAP in your environment, you should definitely check out the Changelog. If you don't... Perhaps this will be a good time to reconsider. I'm starting to like the concept of the directoory being at the center of the infrastructure more and more. It's a good place to share data between hosts in the cluster and integrate their services into a cohesive whole - the imrovements in the SMTP support (nullmailer, Postfix and saslauthd using LDAP for authentication) really show the strengths of this setup. Expect more integration in the future. Also kudos to Rainer Schuth for starting work on integrating Postfix with LDAP; the rest of the mail stack will follow in the next DebOps releases. One issue I can see in the future is the lack of a handy GUI for the LDAP directory itself. Apache Directory Studio is a very useful tool, but it might be a bit too much for day to day operations on the directory. There are some solutions like FusionDirectory or LDAP Account Manager available, but I'm not sure how compatible they will be with the LDAP infrastructure maintained in DebOps. On one hand, having an easy to use GUI would probably be important in the future, on the other picking one up right now might hinder the design of LDAP integration in various roles and applications. Hmmm, decisions, decisions... Revamped online documentation ----------------------------- Big thanks should go to Tasos Alvas for revamping project documentation[3]. Keeping the docs up to date and relevant is a hard but important work, and I'm really grateful for his contribution. I hope that the new documentation structure is easier to find your way around. [3]: https://github.com/debops/debops/pull/1129 Cleanup time ------------ I plan the next development cycle to be shorter, only two months, to shift the release cycle back one month to be better aligned with releases of some other projects like Ubuntu. This should result in DebOps v2.0.0 release around the end of January 2020. Because of that, during this development cycle I plan to focus on cleaning up old code and removing bitrot from the project instead of adding new stuff. Perhaps there will be some time near the end to add something new near the next release, as long as the cleanup is finished. The things I'd like to take care of during cleanup: - The old '[debops_<role>]' Ansible inventory groups will be removed. - Hard role dependencies should be moved from 'meta/main.yml' files to the role tasks, using the 'import_role' Ansible module. Any lingering soft dependencies will be moved to the playbooks. - Old non-namespaced Ansible tags will be removed or replaced with the namespaced ones. - The tasks that use 'dpkg-divert' to divert/revert configuration files should be converted to use the new 'dpkg_divert' custom Ansible module included in DebOps. This will be used as a validation for the module which eventually could be submitted to Ansible core. - Various paths in the lookup('password') lookups used by the 'debops.secret' role should be cleaned up and switched from using 'ansible_fqdn' variable to 'inventory_hostname'. - File-based Ansible local facts in various roles should be converted to Python scripts, roles that don't have a fact script should get one for completeness. - The Python scripts executed via the 'script' Ansible module should be converted to normal Ansible modules. This should solve problems with Python version detection on remote hosts and/or Ansible Controller. - Versions of various upstream applications like Elastic stack, Icinga plugins, etc. should be updated to the latest releases where possible. You can run the 'make versions' command to see the current version selection. Further conversion into Ansible Collection ------------------------------------------ Since Ansible 2.9 release, it seems that the general concept of Ansible Galaxy Collections has been fleshed out, at least in the module and role department. Unfortunately, DebOps monorepo doesn't really fit in this new model, mostly because roles are named using 'debops.*' format which makes the project unusable as an Ansible Collection. Because I would like to see DebOps as a go-to project for Debian-related infrastructure in Ansible, this will have to change. I think that the most correct solution for this problem will be to rename all roles and drop the 'debops.' prefix from them. This will also include changes in the documentation like reference names, updated playbooks and role dependencies, modification in test scripts, and so on. I would like to preserve the ability to use DebOps roles outside of a collection, through the included playbooks. Ansible 2.8 added the 'collections' keyword on the playbook level to faciliate that, but this would mean that the playbooks will be broken on Ansible 2.7 and below, which is included in the current Debian Stable (Buster). At the moment Ansible 2.8 is in Debian Testing (Bullseye), perhaps in a short time a backported version will be available on Buster, which should solve the issue - for the moment users should be able to install Ansible 2.8+ using the upstream APT repository or by building the .deb package locally; 'debops.ansible' role should help with that. I also need to check how the 'roles_path' Ansible configuration variable works with Collections, to see if the old model will still work. One big issue which will remain is creating backports of changes to older DebOps releases through the 'git cherry-pick' command. It looks like the command has some ability to resolve file renames, but I'm not sure yet how much additional work will be involved in backporting the changes. Therefore it is important to make the conversion sooner rather than later to benefit from backports in the future. The new major release (v2.0.0) is also a good fit for such a change, in my opinion. That would be it for the moment. Hopefully the next set of changes will be smooth and without major issues. Until next time, Maciej