[debops-users] Opinions on Ansible role repository layout
by Maciej Delmanowski
This is a cross-post from a mail of one of the DebOps users, I thought that
others might be interested in the topic.
On Feb 08, Nicolas Quiniou-Briand wrote:
> I will start to share Ansible roles for the PacketFence project on Github
> but I wonder what is the best repository layout for Ansible Galaxy : one
> repo with all roles or one repo per role. I notice that on DebOps project,
> all the roles are located under one repo.
>
> I just want to know what are the pros, cons and limitations of this
> repository layout from your point of view.
Hello,
I'm obviously biased, so please keep that in mind.
The choice depends on a few factors. How many roles do you plan to manage at
once? Are they completely independent entities, or do they work together in
some fashion? How many people you have available to maintain them?
You should probably read the mailing list threads in which I talked about
merging DebOps into a single repository:
https://lists.debops.org/pipermail/debops-users/2017-August/000078.html
https://lists.debops.org/pipermail/debops-users/2017-September/000096.html
And here's the afermath of the merge:
https://lists.debops.org/pipermail/debops-users/2017-October/000102.html
I also wrote about this very subject on Reddit:
https://www.reddit.com/r/ansible/comments/9n3p37/develop_roles_with_git_b...
actices/e7k7gza/?context=0
But pros and cons of each model? Let's see...
Pros of separate git repositories:
- it's easier to hand a role or a few of them to somebody else if you don't
want to deal with them anymore
- separate teams can work on one set of roles independently, each one having
access to specific repositories
- it's a simple publishing model, estabilished by Ansible Galaxy, easy to use
with a limited number of roles (10-20 max, I'd say)
Cons of separate git repositories:
- scales horribly unless you write your own toolset to handle large number of
repositories
- lots of overhead when you want to synchronize multiple roles together and
rely on Galaxy to distribute them, because Galaxy uses git tags to handle
versioning. Tags, tags everywhere...
- if you want to use GitHub for issues and pull requests, prepare to deal with
issues and PRs per-repository. There's no good way to centralize it, unless
you create a dummy "issue repository" which only confuses users, and you
still need to deal with PRs per repository anyway.
Pros of a monorepo:
- everything in one place - roles, issues, pull requests, additional tooling,
everything in lockstep. Awesome with a large number of roles.
- you just have to install one git "package" and all of your roles are
available for consumption. You can still use specific roles in playbooks,
there's no downside here.
- git grep over everything. What do you mean, everything? EVERYTHING!
Cons of a monorepo:
- existing tools around Ansible like Molecule are geared towards a single role
per repository model. If you go with a monorepo, you probably will have to
create a custom toolset for it as well, but the problems you have to solve
will be different.
- everything is in lockstep, which might be a problem when you have multiple
teams that want to move at different pace. Scheduling tags/releases based on
time could be a solution to that.
- everyone gets the entire repository / all of the roles. Depending on
preferences or application, that might not be a desirable thing to have.
Ansible Galaxy actually has a very nice support for multi-role repositories
now. I'd say that if you want to start publishing roles from scratch, going
with a multi-role repo is much more benefical. The single Ansible role per
repository was forced on everyone because of limitations of Ansible Galaxy
at the time. It's all better now, go with the monorepo.
Do you mind if I cross-post it to the DebOps mailing list? It might be an
interesting discussion, I wonder how the monorepo looks from the users' side.
Cheers,
Maciej
6 years
[debops-users] problems with debops.netbase role
by Jan Kowalsky
Hi all,
the role debops.netbase gives me the follwoing error:
TASK [debops.netbase : Manage entries in /etc/hosts]
*************************************************************************************************
task path:
/home/jankow/.local/share/debops/debops/ansible/roles/debops.netbase/tasks/main.yml:51
fatal: [test.example.org]: FAILED! => {"msg": "template error while
templating string: no filter named 'parse_kv_config'. String: {{
netbase__combined_hosts | parse_kv_config }}"}
Is this an bug or do I miss something.
Regards
Jan
6 years, 1 month
[debops-users] New DebOps release - v0.8.1
by Maciej Delmanowski
Hello everyone,
The next release of DebOps is here, after almost 6 months. The previous
release was done after 4 months of work, I wonder if a few months between
releases is a good trend.
Also, Groundhog Day!
New DebOps release, v0.8.1
--------------------------
You can find the new version of DebOps on:
GitHub: https://github.com/debops/debops/releases/tag/v0.8.1
PyPI: https://pypi.python.org/pypi/debops/0.8.1
Galaxy: https://galaxy.ansible.com/debops/debops/
You can upgrade the Python-based installation by running the command:
pip install --upgrade debops
DebOps is again oficially available on the Ansible Galaxy website, and the
multi-role repository can be downloaded using the experimental 'mazer' content
manager. You can find more details about installing DebOps in different ways
in the rewritten installation documentation:
https://docs.debops.org/en/v0.8.1/user-guide/install.html
The brief Changelog can also be found on the documentation page:
https://docs.debops.org/en/v0.8.1/news/changelog.html
Complete, detailed changelog can be viewed using the 'git log' command.
The DebOps documentation has a separate page which details important changes
from previous release in the Ansible inventory or on the remote hosts which
you might need to perform manually:
https://docs.debops.org/en/v0.8.1/news/upgrades.html
The Python packages available on PyPI, as well as the tarballs available on
GitHub are signed with my GPG key. You can get it from the OpenPGP keyserver
network using the command:
gpg --keyserver hkp://pool.sks-keyservers.net \
--recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC
The v0.8.1 release ended up with ~400 commits. Here's the list of people
involved in creating it, apart from merge commits:
244 Maciej Delmanowski
24 Rainer 'rei' Schuth
16 Markus Wigge
14 Robin Schneider
12 Evilham
12 Thomas Danielsson
11 Jérémy Gardais
10 Mike Nowak
8 Martin Kimmerle
8 Patryk Ściborek
8 Serge Victor
4 Nicolas Quiniou-Briand
3 Norbert Summer
3 Pedro Luis López Sánchez
2 André Jucovsky Bianchi
2 Eric Renfro
2 Jan Kowalsky
2 Stuart Mumford
1 Alex Schmitz
1 Aloïs Oddon
1 Christian Fischer
1 Fabio Bonelli
1 Gaudenz Steinlin
1 Jack Hill
1 Jonathan Struebel
1 Kamila Součková
1 Konstantinos Koukopoulos
1 Korbinian Preisler
1 Mariska Hoogenboom
1 Nick Janetakis
1 Petr Gajdůšek
1 Russell T. Sherman
1 Stuart Read
Big thanks to everyone involved in this release, for finding out and fixing
bugs, and adding new features to the roles.
Goals reached since latest release
----------------------------------
The Redis support has been redesigned - the 'debops.redis' role has been split
into 'debops.redis_server' and 'debops.redis_sentinel' roles. Both roles
support multi-instance installations.
The IP forwarding configuration was rethought, the 'debops.ifupdown' role now
uses the 'debops.sysctl' role to configure kernel parameters independently for
each network interface. Now the packet forwarding is not configured globally
any more, but independently per interface. Unfortunately some software like
'lxc-net' or 'libvirt' can still enable IP forwarding globally, I haven't
found a workaround for that yet.
The 'debops.dnsmasq' role has been rewritten and now uses the custom DebOps
filter plugins to merge configuration from different sources together. The
role can now synergize with the 'lxc-net' service managed by the 'debops.lxc'
role to allow easy use of the internal LXC containers, which can be reached by
their hostname instead of an IP address.
Unfortunately, nothing happend with the LDAP support in DebOps, but I hope to
improve it in the next release.
Supercharged LXC support
------------------------
During last few months I focused on improving the LXC container support in
DebOps. The 'debops.lxc' role gained support for managing unprivileged LXC
containers, lots of configuration tweaks and a few custom scripts to support
easier container management from the command line.
The 'lxc-net' service is now configured by default to support internal LXC
containers, with their own DNS subdomain. If the 'debops.dnsmasq' role is
configured on the LXC host, containers can be reached using their hostnames
instead of IP addresses - this can be used to give unprivileged users access
to their own LXC containers without giving them access to the 'lxc-*' commands
to not let them mess with each other's containers.
The 'lxc_ssh.py' custom Ansible connection plugin has been included in DebOps.
It can be used to manage LXC containers on the remote host with direct SSH
access. You can find the details in the 'debops.lxc' role documentation.
On hosts with 'systemd', the LXC containers will be started using the
'lxc@.service' instances, with custom overrides. This helps the service
manager correctly shut down the LXC containers at reboot and avoids issues
with forcibly killing them after a timeout.
Reorganization of the bootstrap playbook
----------------------------------------
The initial configuration of the '/etc/hosts' file done by the
'debops.bootstrap' playbook has been ripped from it, redesigned, and
incorporated into the 'debops.netbase' role, which is now included in the
'bootstrap.yml' playbook. This should ensure that the '/etc/hosts'
configuration is idempotent between the bootstrap and common playbooks, and
the domain configuration is set correctly the first time. The 'debops.netbase'
role supports different domain configurations (via external DNS, directly in
'/etc/hosts') and should correclty update the host database in most cases.
The 'debops.apt' role has been added to the 'bootstrap.yml' playbook to allow
early APT configuration. This should ensure that the support for HTTPS APT
repositories, as well as release backports is present when needed by other
roles.
The last thing left in the 'debops.bootstrap' role is management of the System
Administrator accounts. I'm thinking about creating a new role for that
purpose, which would allow configuration of multiple, separate, sysadmin UNIX
accounts as well as support LDAP configuration if present. After that the
'debops.bootstrap' role will probably be removed and replaced completely by
the playbook itself.
Improvements in the DebOps repository
-------------------------------------
The Vagrant environment has been updated - Vagrant hosts now use Avahi to
discover other hosts in the same cluster and configure a custom
'*.vagrant.test' DNS domain in their '/etc/hosts' files. This drops the need
for an external DNS/DHCP server for the test environment, however the setup
requires use of a bridged network. Best results can be achieved under Linux,
with LXC or libvirt environments.
The DebOps roles and playbooks have been cleaned up and are now tested using
the 'ansible-lint' command. This results in a perfect 5/5 score on the Ansible
Galaxy, woohoo! ;-)
The 'make versions' command can be used to check currently pinned versions of
third-party software used in DebOps roles, and compare them to the upstream
versions of the released software. Since the project gains support for more
third-party applications installed via 'git' repositories, this should help
the project track the updated software better. The roles that use the software
from the Debian Archive, or third-party APT repositories, are not included in
these checks. The check has been implemented using the 'uscan' command
included in the Debian 'devscripts' APT package.
The 'debops.ansible_plugins' role now contains all of the custom Ansible
plugins used in DebOps roles (apart from the connection and callback plugins
which are not useful on the role level). This should fix issues with the
custom lookup plugins not being available when DebOps roles are used
independently from the whole DebOps project. The lookup plugins don't require
the 'debops' Python package to be installed, and will work just fine without
it,
Plans for the next release
--------------------------
At the moment, Debian Testing release is beginning to freeze, with the Debian
Buster release aimed some time around August 2019. If we keep with the 6-month
release cycle, the next DebOps release could be done somewhere around the same
time. So let's aim for that and see what happens. Some of the new Debian
Buster features like AppArmor enabled by default will have to be implemented
in DebOps roles, that probably will end up as one of the main goals for the
next DebOps version.
In the next release cycle I would like to focus on the clustered environment
managed by DebOps - DNS (bind9), LDAP (slapd, sssd, some kind of LDAP
management interface like LAM, better support for LDAP in various
applications), remote storage (NFSv4, block storage provided by Minio, Docker
Registry for container storage), centralization of various services
(centralized logs via logstash, filebeat, load balancing via haproxy and/or
keepalived). I'm interested in further exploring the way storage and compute
nodes can be managed together, to allow easy reinstallation or scaling of
compute nodes as well as data replication and security on the storage nodes.
I'd like to explore the "unprivileged" side of DebOps more, as explained in my
previous e-mail. At present it seems that the roles could be designed with
usage via an admin account by default, but allow for direct unprivileged
operation in some cases. I'm not sure how the main playbook will have to
change to accomodate more unprivileged operation yet.
Once again, thanks to everyone involved in this project, and remember to have
fun with it. See you next time.
Maciej Delmanowski
6 years, 1 month
