Re: [debops-users] debops-users Digest, Vol 24, Issue 1
by Julien PEETERS
Hi Maciej, hi all contributors,
I hope you are well.
What a fantastic work you have done all together. Very nice!
The integration of MinIO motivates me to go back on the glusterfs role.
See you soon !
Best,
Julien
On 02/12/2019 12:00, debops-users-request(a)lists.debops.org wrote:
> Send debops-users mailing list submissions to
> debops-users(a)lists.debops.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.debops.org/mailman/listinfo/debops-users
> or, via email, send a message with subject or body 'help' to
> debops-users-request(a)lists.debops.org
>
> You can reach the person managing the list at
> debops-users-owner(a)lists.debops.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of debops-users digest..."
>
>
> Today's Topics:
>
> 1. New DebOps stable release - v1.2.0 (Maciej Delmanowski)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 1 Dec 2019 22:42:28 +0100
> From: Maciej Delmanowski <drybjed(a)drybjed.net>
> To: debops-users(a)lists.debops.org
> Subject: [debops-users] New DebOps stable release - v1.2.0
> Message-ID: <20191201214228.bkglbtji7sravq4a(a)drybjed.net>
> Content-Type: text/plain; charset="utf-8"
>
> Hello everyone,
>
> The next DebOps release is here! I kind of forgot that December 1st is on
> Sunday and not November 31th, but I hope it's close enough.
>
>
> New DebOps release, v1.2.0
> --------------------------
>
> You can find the new version of DebOps on:
>
> GitHub: https://github.com/debops/debops/releases/tag/v1.2.0
> PyPI: https://pypi.python.org/pypi/debops/1.2.0
> Galaxy: https://galaxy.ansible.com/debops/debops/ (but see below)
>
> You can upgrade the Python-based installation by running the command:
>
> pip install --upgrade debops
>
> The support for Galaxy Collections has been improved, but there are still
> issues - namely, Galaxy does not support role dependencies properly[1], and
> because of that the 'namespace.project.role' role naming scheme cannot be used
> in the playbooks yet. Installation via PyPI package or directly from GitHub
> repository should be preferred this time around.
>
> Installation instructions can be found here:
>
> https://docs.debops.org/en/stable-1.2/user-guide/install.html
>
> The brief Changelog can also be found on the documentation page:
>
> https://docs.debops.org/en/stable-1.2/news/changelog.html
>
> Complete, detailed changelog can be viewed using the 'git log' command. You can
> use the 'git log --no-merges' command to skip the "boring" merge commits.
>
> The DebOps documentation has a separate page which details important changes
> from previous release in the Ansible inventory or on the remote hosts which
> you might need to perform manually:
>
> https://docs.debops.org/en/stable-1.2/news/upgrades.html
>
> The Python packages available on PyPI, as well as the tarballs available on
> GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver
> network using the command:
>
> gpg --keyserver hkp://pool.sks-keyservers.net \
> --recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC
>
> Since the v1.1.0 release in August 2019, there were ~504 commits in the DebOps
> repository, not counting merge commits. Here's the breakdown of the committers
> in the v1.2.0 release:
>
> 306 Maciej Delmanowski
> 108 Rainer 'rei' Schuth
> 21 Robin Schneider
> 15 Tasos Alvas
> 13 Imre Jonk
> 13 Nicolas Quiniou-Briand
> 8 Leonardo Bechea
> 3 Douglas Heriot
> 3 Hartmut Goebel
> 3 Thomas Finstad Larsen
> 2 Aljosha Papsch
> 2 Gaudenz Steinlin
> 1 Dionysis Grigoropoulos
> 1 Hendrik Visage
> 1 Hüseyin Uslu
> 1 Patryk Ściborek
> 1 Rolf Morgenstern
> 1 Rostislav Kandilarov
> 1 Thomas Danielsson
>
> Thanks to everyone involved for helping shape up this project, and see you in
> the commits.
>
> [1]: https://github.com/ansible/galaxy/issues/1938
>
>
> Goals reached since previous release
> ------------------------------------
>
> As usual, only some of the previous release goals[2] managed to get done. The
> redesigned 'debops.golang' role is one of them, and new roles for MinIO and
> MinIO Client (mcli) use it to install the upstream applications. More Go-based
> applications will follow in the future.
>
> The roles for OpenNebula didn't make it this time, and with the current plans
> I'll probably leave this for after v2.0.0 release in January. The same goes
> for improved Samba support - unfortunately I didn't get to updating the
> 'debops.samba' role yet, or even integrating it with LDAP. It will have to
> wait. There's also no .deb package support at the moment, perhaps in the next
> release.
>
> [2]: https://lists.debops.org/pipermail/debops-users/2019-August/000243.html
>
>
> Reshaping the project around LDAP
> ---------------------------------
>
> During this development cycle, lots of work happened in regards to the LDAP
> directory support. The initial directoory layout was redesigned around Groups
> and Roles, Access Control List was revamped a bit to support hiding LDAP
> objects, 6 DebOps roles have been either converted to use the new
> 'debops.ldap' infrastructure, or LDAP support has been added to them... There
> were many changes in the 'debops.slapd' role as well, from new LDAP schema
> installed by default, to support for ACL rule tests. If you use LDAP in your
> environment, you should definitely check out the Changelog. If you don't...
> Perhaps this will be a good time to reconsider.
>
> I'm starting to like the concept of the directoory being at the center of the
> infrastructure more and more. It's a good place to share data between hosts in
> the cluster and integrate their services into a cohesive whole - the
> imrovements in the SMTP support (nullmailer, Postfix and saslauthd using LDAP
> for authentication) really show the strengths of this setup. Expect more
> integration in the future. Also kudos to Rainer Schuth for starting work on
> integrating Postfix with LDAP; the rest of the mail stack will follow in the
> next DebOps releases.
>
> One issue I can see in the future is the lack of a handy GUI for the LDAP
> directory itself. Apache Directory Studio is a very useful tool, but it might
> be a bit too much for day to day operations on the directory. There are some
> solutions like FusionDirectory or LDAP Account Manager available, but I'm not
> sure how compatible they will be with the LDAP infrastructure maintained in
> DebOps. On one hand, having an easy to use GUI would probably be important in
> the future, on the other picking one up right now might hinder the design of
> LDAP integration in various roles and applications. Hmmm, decisions,
> decisions...
>
>
> Revamped online documentation
> -----------------------------
>
> Big thanks should go to Tasos Alvas for revamping project documentation[3].
> Keeping the docs up to date and relevant is a hard but important work, and I'm
> really grateful for his contribution. I hope that the new documentation
> structure is easier to find your way around.
>
> [3]: https://github.com/debops/debops/pull/1129
>
>
> Cleanup time
> ------------
>
> I plan the next development cycle to be shorter, only two months, to shift the
> release cycle back one month to be better aligned with releases of some other
> projects like Ubuntu. This should result in DebOps v2.0.0 release around the
> end of January 2020.
>
> Because of that, during this development cycle I plan to focus on cleaning up
> old code and removing bitrot from the project instead of adding new stuff.
> Perhaps there will be some time near the end to add something new near the
> next release, as long as the cleanup is finished.
>
> The things I'd like to take care of during cleanup:
>
> - The old '[debops_<role>]' Ansible inventory groups will be removed.
>
> - Hard role dependencies should be moved from 'meta/main.yml' files to the
> role tasks, using the 'import_role' Ansible module. Any lingering soft
> dependencies will be moved to the playbooks.
>
> - Old non-namespaced Ansible tags will be removed or replaced with the
> namespaced ones.
>
> - The tasks that use 'dpkg-divert' to divert/revert configuration files should
> be converted to use the new 'dpkg_divert' custom Ansible module included
> in DebOps. This will be used as a validation for the module which eventually
> could be submitted to Ansible core.
>
> - Various paths in the lookup('password') lookups used by the 'debops.secret'
> role should be cleaned up and switched from using 'ansible_fqdn' variable to
> 'inventory_hostname'.
>
> - File-based Ansible local facts in various roles should be converted to
> Python scripts, roles that don't have a fact script should get one for
> completeness.
>
> - The Python scripts executed via the 'script' Ansible module should be
> converted to normal Ansible modules. This should solve problems with Python
> version detection on remote hosts and/or Ansible Controller.
>
> - Versions of various upstream applications like Elastic stack, Icinga
> plugins, etc. should be updated to the latest releases where possible. You
> can run the 'make versions' command to see the current version selection.
>
>
> Further conversion into Ansible Collection
> ------------------------------------------
>
> Since Ansible 2.9 release, it seems that the general concept of Ansible Galaxy
> Collections has been fleshed out, at least in the module and role department.
> Unfortunately, DebOps monorepo doesn't really fit in this new model, mostly
> because roles are named using 'debops.*' format which makes the project
> unusable as an Ansible Collection. Because I would like to see DebOps as
> a go-to project for Debian-related infrastructure in Ansible, this will have
> to change.
>
> I think that the most correct solution for this problem will be to rename all
> roles and drop the 'debops.' prefix from them. This will also include changes
> in the documentation like reference names, updated playbooks and role
> dependencies, modification in test scripts, and so on.
>
> I would like to preserve the ability to use DebOps roles outside of
> a collection, through the included playbooks. Ansible 2.8 added the
> 'collections' keyword on the playbook level to faciliate that, but this would
> mean that the playbooks will be broken on Ansible 2.7 and below, which is
> included in the current Debian Stable (Buster). At the moment Ansible 2.8 is
> in Debian Testing (Bullseye), perhaps in a short time a backported version
> will be available on Buster, which should solve the issue - for the moment
> users should be able to install Ansible 2.8+ using the upstream APT repository
> or by building the .deb package locally; 'debops.ansible' role should help
> with that. I also need to check how the 'roles_path' Ansible configuration
> variable works with Collections, to see if the old model will still work.
>
> One big issue which will remain is creating backports of changes to older
> DebOps releases through the 'git cherry-pick' command. It looks like the
> command has some ability to resolve file renames, but I'm not sure yet how
> much additional work will be involved in backporting the changes. Therefore it
> is important to make the conversion sooner rather than later to benefit from
> backports in the future. The new major release (v2.0.0) is also a good fit for
> such a change, in my opinion.
>
> That would be it for the moment. Hopefully the next set of changes will be
> smooth and without major issues.
>
> Until next time,
> Maciej
>
4 years, 10 months
[debops-users] New DebOps stable release - v1.2.0
by Maciej Delmanowski
Hello everyone,
The next DebOps release is here! I kind of forgot that December 1st is on
Sunday and not November 31th, but I hope it's close enough.
New DebOps release, v1.2.0
--------------------------
You can find the new version of DebOps on:
GitHub: https://github.com/debops/debops/releases/tag/v1.2.0
PyPI: https://pypi.python.org/pypi/debops/1.2.0
Galaxy: https://galaxy.ansible.com/debops/debops/ (but see below)
You can upgrade the Python-based installation by running the command:
pip install --upgrade debops
The support for Galaxy Collections has been improved, but there are still
issues - namely, Galaxy does not support role dependencies properly[1], and
because of that the 'namespace.project.role' role naming scheme cannot be used
in the playbooks yet. Installation via PyPI package or directly from GitHub
repository should be preferred this time around.
Installation instructions can be found here:
https://docs.debops.org/en/stable-1.2/user-guide/install.html
The brief Changelog can also be found on the documentation page:
https://docs.debops.org/en/stable-1.2/news/changelog.html
Complete, detailed changelog can be viewed using the 'git log' command. You can
use the 'git log --no-merges' command to skip the "boring" merge commits.
The DebOps documentation has a separate page which details important changes
from previous release in the Ansible inventory or on the remote hosts which
you might need to perform manually:
https://docs.debops.org/en/stable-1.2/news/upgrades.html
The Python packages available on PyPI, as well as the tarballs available on
GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver
network using the command:
gpg --keyserver hkp://pool.sks-keyservers.net \
--recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC
Since the v1.1.0 release in August 2019, there were ~504 commits in the DebOps
repository, not counting merge commits. Here's the breakdown of the committers
in the v1.2.0 release:
306 Maciej Delmanowski
108 Rainer 'rei' Schuth
21 Robin Schneider
15 Tasos Alvas
13 Imre Jonk
13 Nicolas Quiniou-Briand
8 Leonardo Bechea
3 Douglas Heriot
3 Hartmut Goebel
3 Thomas Finstad Larsen
2 Aljosha Papsch
2 Gaudenz Steinlin
1 Dionysis Grigoropoulos
1 Hendrik Visage
1 Hüseyin Uslu
1 Patryk Ściborek
1 Rolf Morgenstern
1 Rostislav Kandilarov
1 Thomas Danielsson
Thanks to everyone involved for helping shape up this project, and see you in
the commits.
[1]: https://github.com/ansible/galaxy/issues/1938
Goals reached since previous release
------------------------------------
As usual, only some of the previous release goals[2] managed to get done. The
redesigned 'debops.golang' role is one of them, and new roles for MinIO and
MinIO Client (mcli) use it to install the upstream applications. More Go-based
applications will follow in the future.
The roles for OpenNebula didn't make it this time, and with the current plans
I'll probably leave this for after v2.0.0 release in January. The same goes
for improved Samba support - unfortunately I didn't get to updating the
'debops.samba' role yet, or even integrating it with LDAP. It will have to
wait. There's also no .deb package support at the moment, perhaps in the next
release.
[2]: https://lists.debops.org/pipermail/debops-users/2019-August/000243.html
Reshaping the project around LDAP
---------------------------------
During this development cycle, lots of work happened in regards to the LDAP
directory support. The initial directoory layout was redesigned around Groups
and Roles, Access Control List was revamped a bit to support hiding LDAP
objects, 6 DebOps roles have been either converted to use the new
'debops.ldap' infrastructure, or LDAP support has been added to them... There
were many changes in the 'debops.slapd' role as well, from new LDAP schema
installed by default, to support for ACL rule tests. If you use LDAP in your
environment, you should definitely check out the Changelog. If you don't...
Perhaps this will be a good time to reconsider.
I'm starting to like the concept of the directoory being at the center of the
infrastructure more and more. It's a good place to share data between hosts in
the cluster and integrate their services into a cohesive whole - the
imrovements in the SMTP support (nullmailer, Postfix and saslauthd using LDAP
for authentication) really show the strengths of this setup. Expect more
integration in the future. Also kudos to Rainer Schuth for starting work on
integrating Postfix with LDAP; the rest of the mail stack will follow in the
next DebOps releases.
One issue I can see in the future is the lack of a handy GUI for the LDAP
directory itself. Apache Directory Studio is a very useful tool, but it might
be a bit too much for day to day operations on the directory. There are some
solutions like FusionDirectory or LDAP Account Manager available, but I'm not
sure how compatible they will be with the LDAP infrastructure maintained in
DebOps. On one hand, having an easy to use GUI would probably be important in
the future, on the other picking one up right now might hinder the design of
LDAP integration in various roles and applications. Hmmm, decisions,
decisions...
Revamped online documentation
-----------------------------
Big thanks should go to Tasos Alvas for revamping project documentation[3].
Keeping the docs up to date and relevant is a hard but important work, and I'm
really grateful for his contribution. I hope that the new documentation
structure is easier to find your way around.
[3]: https://github.com/debops/debops/pull/1129
Cleanup time
------------
I plan the next development cycle to be shorter, only two months, to shift the
release cycle back one month to be better aligned with releases of some other
projects like Ubuntu. This should result in DebOps v2.0.0 release around the
end of January 2020.
Because of that, during this development cycle I plan to focus on cleaning up
old code and removing bitrot from the project instead of adding new stuff.
Perhaps there will be some time near the end to add something new near the
next release, as long as the cleanup is finished.
The things I'd like to take care of during cleanup:
- The old '[debops_<role>]' Ansible inventory groups will be removed.
- Hard role dependencies should be moved from 'meta/main.yml' files to the
role tasks, using the 'import_role' Ansible module. Any lingering soft
dependencies will be moved to the playbooks.
- Old non-namespaced Ansible tags will be removed or replaced with the
namespaced ones.
- The tasks that use 'dpkg-divert' to divert/revert configuration files should
be converted to use the new 'dpkg_divert' custom Ansible module included
in DebOps. This will be used as a validation for the module which eventually
could be submitted to Ansible core.
- Various paths in the lookup('password') lookups used by the 'debops.secret'
role should be cleaned up and switched from using 'ansible_fqdn' variable to
'inventory_hostname'.
- File-based Ansible local facts in various roles should be converted to
Python scripts, roles that don't have a fact script should get one for
completeness.
- The Python scripts executed via the 'script' Ansible module should be
converted to normal Ansible modules. This should solve problems with Python
version detection on remote hosts and/or Ansible Controller.
- Versions of various upstream applications like Elastic stack, Icinga
plugins, etc. should be updated to the latest releases where possible. You
can run the 'make versions' command to see the current version selection.
Further conversion into Ansible Collection
------------------------------------------
Since Ansible 2.9 release, it seems that the general concept of Ansible Galaxy
Collections has been fleshed out, at least in the module and role department.
Unfortunately, DebOps monorepo doesn't really fit in this new model, mostly
because roles are named using 'debops.*' format which makes the project
unusable as an Ansible Collection. Because I would like to see DebOps as
a go-to project for Debian-related infrastructure in Ansible, this will have
to change.
I think that the most correct solution for this problem will be to rename all
roles and drop the 'debops.' prefix from them. This will also include changes
in the documentation like reference names, updated playbooks and role
dependencies, modification in test scripts, and so on.
I would like to preserve the ability to use DebOps roles outside of
a collection, through the included playbooks. Ansible 2.8 added the
'collections' keyword on the playbook level to faciliate that, but this would
mean that the playbooks will be broken on Ansible 2.7 and below, which is
included in the current Debian Stable (Buster). At the moment Ansible 2.8 is
in Debian Testing (Bullseye), perhaps in a short time a backported version
will be available on Buster, which should solve the issue - for the moment
users should be able to install Ansible 2.8+ using the upstream APT repository
or by building the .deb package locally; 'debops.ansible' role should help
with that. I also need to check how the 'roles_path' Ansible configuration
variable works with Collections, to see if the old model will still work.
One big issue which will remain is creating backports of changes to older
DebOps releases through the 'git cherry-pick' command. It looks like the
command has some ability to resolve file renames, but I'm not sure yet how
much additional work will be involved in backporting the changes. Therefore it
is important to make the conversion sooner rather than later to benefit from
backports in the future. The new major release (v2.0.0) is also a good fit for
such a change, in my opinion.
That would be it for the moment. Hopefully the next set of changes will be
smooth and without major issues.
Until next time,
Maciej
4 years, 10 months
