New stable DebOps release: v2.1.0

Hello everyone, It's time for a new DebOps release, this time with a sliiight delay due to various factors. But it's here! New DebOps release, v2.1.0 -------------------------- You can find the new version of DebOps on: GitHub: https://github.com/debops/debops/releases/tag/v2.1.0 PyPI: https://pypi.python.org/pypi/debops/2.1.0 Galaxy: https://galaxy.ansible.com/debops/debops/ You can upgrade the Python-based installation by running the command: pip3 install --upgrade debops Installation instructions can be found here: https://docs.debops.org/en/stable-2.1/introduction/install.html The brief Changelog can also be found on the documentation page: https://docs.debops.org/en/stable-2.1/news/changelog.html Complete, detailed changelog can be viewed using the 'git log' command. You can use the 'git log --no-merges' command to skip the "boring" merge commits. The DebOps documentation has a separate page which details important changes from previous release in the Ansible inventory or on the remote hosts which you might need to perform manually: https://docs.debops.org/en/stable-2.1/news/upgrades.html The Python packages available on PyPI, as well as the tarballs available on GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver network using the command: gpg --keyserver hkp://pool.sks-keyservers.net \ --recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC This release was meant to be done in April 2020. Unfortunately for various reasons it didn't happen, sorry about that. Due to the delay, the release schedule will be changed a bit to accomodate the mid-development cycle release; the next one will be in October 2020. The v2.1.0 development cycle finished with 490 commits (not counting merges) which is pretty average for the project. Here's a breakdown of committers this time around: 340 Maciej Delmanowski 81 Robin Schneider 19 Pedro Luis López Sánchez 10 Imre Jonk 7 Gabriel Lewertowski 6 Rainer 'rei' Schuth 6 Tasos Alvas 5 Hendrik Visage 2 Alin Alexandru 1 Ant Somers 1 Bao Nguyen 1 Hanno - Felix Wagner 1 Jean Sorgemoel 1 Jonathan Struebel 1 Jérémy Gardais 1 Korbinian Preisler 1 Patryk Ściborek 1 Paulo da Silva 1 Sandro Jäckel 1 Serge Victor 1 Steffen Lindner 1 Thiago Talma 1 Thomas Danielsson Thanks to everyone involved, and see you in the commits. Goals reached since previous release ------------------------------------ As usual, only part of the plans for the next DebOps release[1] was finished before the next one. Let's see what goals were reached: - New DebOps roles - 7 new roles were added to DebOps (6 if you count InfluxDB client/server together). That's less than I hoped, but most of that is on me not doing reviews of the pull requests. I'll try to do better next time. - The "wrapper roles" for firewall, NTP and webserver services didn't make the cut eiher, but I still plan to introduce that concept in the future. Stay tuned. - Refresh of the Mailman role - done! You probably are reading this on the new Mailman 3 instance that manages DebOps mailing lists. Unfortunately, changes in the application broke some of the mailing list URLs referenced in older e-mail messages, but hopefully the new search functionality can mitigate this. - Support for ELK stack and logging - partially done. I didn't get around to writing the Logstash role, but @PedroLuisLopez started working on one[2]. If you are interested in that service, check his PR out and leave a review. What I did do is add the 'journald' Ansible role and refresh the 'rsyslog' role, so at least the start of the logging pipeline should be modernized. [1]: https://lists.debops.org/hyperkitty/list/debops-users@lists.debops.org/th... [2]: https://github.com/debops/debops/pull/1370 The 'stable-1.0' release is retired ----------------------------------- DebOps v1.0.0 was released on 22th May 2019 and had 18 patch releases with about ~390 commits; most of these were backports from later releases. Major highlights of this release included introduction of a redesigned LDAP support project-wide, role for Docker Registry that integrated with GitLab and removal of the 'bootstrap' role - its functionality was moved to various other roles. It was also the first release published on the new Ansible Galaxy as an Ansible Collection. There will be no new updates to the 'stable-1.0' branch. Existing users are encouraged to upgrade to newer DebOps releases, and to read the upgrade notes for information about needed changes. The next DebOps release to be retired will be 'stable-1.1', on 25th August 2020. Additionally, the 'stable-1.1' and 'stable-1.2' releases will not be uploaded to Ansible Galaxy anymore due to too many changes needed to make them compatible with Galaxy requirements. These branches will be retired in the next few months, so that shouldn't be a problem; users of the old branches should switch to the 'stable-2.0' branch or newer. New release highlights ---------------------- There were many changes in the project since the v2.0.0 release, many of them were backported to the previous releases as well. Here are some highlights from the Changelog: - Starting with the v2.1.0 release, the DebOps project is now compliant with the REUSE Specification[3] which defines how Open Source projects should maintain Copyright and License information. All files in the DebOps repository have been audited, copyright information and licensing is up to date and the compliance is checked[4] on Travis-CI before any pull requests are merged. - The annoying "Could not match supplied host pattern" warning messages during Ansible playbook execution will be suppressed via Ansible configuration options. Existing deployments have to be modified manually to handle this, though. - Support for Ansible Collections is much better now, bugs with wrong collection references in Ansible plugins have been ironed out. The DebOps Collection has been split into separate smaller collections due to the artifact size limits imposed by Ansible Galaxy service. Only the main collection ('debops.debops') is mandatory, others have only additional roles. Installation of role collections should be handled by the 'ansible-galaxy' tool using collection dependencies. - Ansible now handles undefined nested dictionaries cleanly so most variable definitions like: var: '{{ ansible_local.core.admin_users if (ansible_local|d() and ansible_local.core|d() and ansible_local.core.admin_users|d()) else [] }}' have been reduced to: var: '{{ ansible_local.core.admin_users | d([]) }}' Thanks go to @ypid for handling the brunt of the work updating these! [3]: https://reuse.software/ [4]: https://api.reuse.software/info/github.com/debops/debops Third party code vendoring and Ansible Collections -------------------------------------------------- During previous release I mentioned an issue with use of the third-party code in DebOps via Ansible Collections or some kind of vendoring solution. After thinking about this issue for some time, I came to the conclusion that DebOps needs to be fully converted to an Ansible Collection model instead of modifying the existing codebase in the collection build script to conform to the requirements of Collections. With this approach, DebOps will be fully compatible with other Ansible Collections in all three usage models (git monorepo, Python package, Ansible Collection). DebOps roles will be able to reference roles, modules and other Ansible plugins in third-party Ansible Collections. At the moment the Collections don't have any kind of GPG-based verification possibility, I'm not sure how to solve that effectively. This of course will require some changes in the monorepo like introduction of 'ansible/collections/ansible_collections/debops/debops' symlink that points to the 'ansible/' directory, and the file structure in the Python package will have to be rearranged, but these are minor details. The 'ansible_plugins' role will also become unnecessary since Collections can natively resolve custom plugins and modules when they are specified using FQCN names. Another thing that will have to be done before the changes are implemented is migration of all 'debops-contrib.*' roles to the main namespace. This has to happen to allow them to be used in Ansible Collection model. The breaking change will happen in the playbooks and roles maintained by DebOps users. The playbooks and roles need to include the 'collections:' keyword to tell Ansible where to look for roles and plugins; in the roles themselves references to filter, lookup and other plugins will have to be modified as well. The changes to roles needed to make them collection-compatible will break the "legacy" usage as standalone Ansible roles, so the change needs to happen everywhere at once, aka "flag day"[5]. This will be a major change, so the next DebOps release will be v3.0.0. Longer release cycle this time will hopefully help with development and release on time. Since the change will happen on the 'master' branch at some point, if you don't want to be affected by it unexpectedly, you might consider sticking to the 'stable-2.1' branch for the next development cycle. [5]: https://en.wikipedia.org/wiki/Flag_day_(computing) Redesign of the 'debops*' scripts and project directories --------------------------------------------------------- The 'debops*' scripts will also have to be updated to support the new Ansible Container-based model. This is a good opportunity to clean up and redesign them, which I plan to do in the coming months. Some of the things I would like to implement: - change all 'debops-*' commands to be subcommands and implemented in Python instead of bash where necessary - add support for multiple Ansible inventories in one "project" directory with one inventory being a primary one. Inventories can then be treated as "views", where the primary inventory manages the system itself, additional inventories are configured to use unprivileged user accounts to deploy applications, etc. - implement better management subcommands on the Ansible Controller to maintain project directories. Since DebOps is used from different OS platforms (Linux/macOS/WSL/FreeBSD), keeping it cross-platform should be easy with Python. - add support for a Bridge CA to allow different environments to talk to each other securely over TLS. This would also allow for easier initialization and managemnt of the internal Certificate Authorities on the Ansible Controller without an awkward setup by running the 'pki' role against a host. - native support for project directories stored in git repositories, with separate git repository for each Ansible inventory. This would enable sharing the projects (we really need a better name for that concept) between users in a distributed environment with tracked and synced changes. The 'debops' scripts could automatically commit changes, check remote repositories for updates, and so on. - add support for runtime synchronization and communication between 'debops' execution processes via external services like RabbitMQ, MQTT, and similar. This would simplify coordination between system administrators and other users in distributed setups. It will be a good opportunity to learn some good Python practices for me. For inspiration I plan to look to AWX project as well as FreeIPA, although focused on usage via CLI and more hands-on approach. DebOps is good in bootstrapping new environments, and I would like to keep it that way. If you have any more interesting ideas that could be added to the scripts after the refactor is complete, let me know. Plans for the next development cycle ------------------------------------ Redesign of the 'debops' scripts and switch to the Ansible Collection model will probably take significant amount of time in the next development cycle, so I don't plan to introduce large changes in the existing DebOps roles. However what I want to do is add support for the 'firewalld' service and improve firewall configuration throught the project, so hopefully there will be time to implement this before October. Till the next time, Maciej