Hello everyone, Due to the discussion in another thread about switching from nginx to Caddy as the webserver[1], I'd like to pick your brains about dealing with shared but interchangeable resources in DebOps. Currently, infrastructure managed by DebOps is defined in separate Ansible playbooks, each playbook uses one or multiple Ansible roles. Some of these roles manage resources (services, files, configuration, etc.) that are singular in nature - there's only one set of APT preferences available to be configured, or one PostgreSQL database service, and so on. There's no need to think aboout alternatives for these. One the other end, there are usually application roles like gitlab, netbox, basically any higher-level application that rely on the lower level Ansible roles to configure other services for them, like firewall, web server, and so on. In most cases these application roles/playbooks are not enabled by default so users can mix and match them in their infrastructure at will, and there's no need to worry about shared interface for them. But in the middle between these two extremes, there's a set of services that can be shared between different applications and are provided in Debian with alternatives which influence the user preference for them. Some examples include: - webserver (nginx, apache2, caddy) - firewall (ferm, firewalld, ufw) - SMTP server (nullmailer, postfix, dma) - NTP server (openntpd, chrony, timesyncd) Due to how DebOps codebase is designed and restrictions around it (read only roles and playbooks), it is hard to come up with a good method of handling these services via Ansible. Over the years, we found different strategies of dealing with them in the project: 1. Create a role that manages a service and can support alternative applications to do so. This is most prominently used in the 'ntp' role, which supports installation of different NTP servers chosen by the user. It's a reasonable method of encapsulation for this kind of problem, but it makes the role much more complicated that it needs to be, and is not sustainable with more complex services like webservers. 2. Create two or more alternative playbooks which use alternative services, each playbook is activated by its own Ansible inventory group. This happened with the 'owncloud' role and its possibility to use either 'nginx' or 'apache2' as the webserver frontend. This method moves the problem from the role level all the way to the actual hosts themselves - if somebody wants to install Nextcloud with Apache2 as the frontend for some reason, they cannot use the same host to configure another application that only knows how to integrate with 'nginx', say NetBox, because the Nextcloud installation will be broken in the process. This solution is also not really scalable - if we go one level down, we get the firewall role (ferm) which is used by dozens of other roles. If we add an alternatve 'firewalld' role (which I would like to do at some point), each service that relies on 'ferm' would have two or more alternative playbooks. In an extreme scenario, DebOps could provide users with multiple sets of 'site.yml' playbook trees to choose from, but with each choice the number of those sets would increase exponentially (firewall * webserver, for example). These are two extremes we came up so far, so I suspect that the answer might be somewhere in the middle. A possible solution I came up with is a concept of a "wrapper role" which moves the complexity for managing multiple services in a single role onto itself by importing selected service role during Ansible execution. This solution has interesting properties and drawbacks, but let's start with an example. Below you can find a description of the 'firewall' role. It lets us switch between 'ferm' and 'firewalld' services on the playbook level without the need to modify existing playbooks: # ansible/roles/firewall/defaults/main.yml firewall__enabled: True firewall__service: 'ferm' # alt: firewalld firewall__ferm__dependent_rules: [] firewall__firewalld__dependent_rules: [] # ansible/roles/firewall/tasks/main.yml - name: Manage the firewall service import_role: name: '{{ firewall__service }}' tags: - 'role::{{ firewall__service }}' - 'skip::{{ firewall__service }}' vars: ferm__dependent_rules: '{{ firewall__ferm__dependent_rules }}' firewalld__dependent_rules: '{{ firewall__firewalld__dependent_rules }}' when: firewall__enabled|bool # ansible/playbooks/service/firewall.yml - name: Manage the host firewall hosts: [ 'debops_all_hosts', 'debops_service_firewall' ] become: True roles: - role: firewall tags: [ 'role::firewall', 'skip::firewall' ] With this setup, the complexity is moved "in the middle", into a separate role which then imports the desired final service role. The problem comes with selecting the actual service role - this cannot be done on the Ansible inventory level. The 'import_role' tasks are processed by Ansible before the inventory is even touched, so only the default value ('ferm') would have any effect on the 'import_role' task. But even if we could define that on the inventory level, Ansible would want to run two competing roles on different hosts at the same time. It cannot do that though, and will select the first service role that it knows about (ferm) and run it on all hosts. However, this can be solved by the use of the '--extra-vars' Ansible option. If we specify the 'firewall__service' variable on the command line, we can override it during Ansible execution and change the default 'ferm' role to 'firewalld'. With this in mind, I added the support for global variables in previous release of DebOps[2]. Users can deal with use of different firewall services on different hosts by creating separate inventories for different sets of hosts and selecting different firewall services per inventory. With that in mind, I want to add support for multiple Ansible inventories in a DebOps project directory with shared resources like PKI infrastructure to faciliate easier environment integration. Other roles that depend on the hypothetical 'firewall' role would just add it in their playbooks as normal and pass the '*__dependent_rules' variables to it with the desired configuration. As long as the dependent configuration does not involve variables from the role defaults and/or is properly wrapped in default([]) constructs, there shouldn't be any issues with this usage. This becomes a bit more complex when we consider other such "wrapper role" that might require dependencies on our "firewall" role. For example, the 'ntp' role would become a wrapper role for 'openntpd', 'chrony', 'ntpdate', 'timesyncd' roles, some of which require firewall configuration, and some not. How can we deal with this? By multiple role imports, of course. Below is an example rewritten 'ntp' role that supports different NTP servers, some defining their own firewall rules: # ansible/roles/ntp/defaults/main.yml ntp__service: 'openntpd' # alt: chrony, timesyncd, ntpdate ntp__deploy_state: 'present' ntp__ferm__dependent_rules: - '{{ openntpd__ferm__dependent_rules | d([]) }}' - '{{ chrony__ferm__dependent_rules | d([]) }}' ntp__firewalld__dependent_rules: - '{{ openntpd__firewalld__dependent_rules | d([]) }}' - '{{ chrony__firewalld__dependent_rules | d([]) }}' # ansible/roles/ntp/tasks/main.yml - name: Run the firewall role import_role: name: 'firewall' vars: firewall__ferm__dependent_rules: - '{{ ntp__ferm__dependent_rules }}' firewall__firewalld__dependent_rules: - '{{ ntp__firewalld__dependent_rules }}' tags: - 'role::firewall' - 'skip::firewall' when: ntp__service in [ 'openntpd', 'chrony' ] - name: Run the service role import_role: name: '{{ ntp__service }}' tags: - 'role::{{ ntp__service }}' - 'skip::{{ ntp__service }}' vars: openntpd__deploy_state: '{{ ntp__deploy_state }}' chrony__deploy_state: '{{ ntp__deploy_state }}' timesyncd__deploy_state: '{{ ntp__deploy_state }}' ntpdate__deploy_state: '{{ ntp__deploy_state }}' # ansible/playbooks/service/ntp.yml - name: Manage the NTP service hosts: [ 'debops_all_hosts', 'debops_service_ntp' ] become: True roles: - role: ntp tags: [ 'role::ntp', 'skip::ntp' ] As you can see, the complexity is removed from the playbook level and encapsulated in the intermediate "wrapper" role. The final service roles can focus on their own services without the need to concern themselves with the alternatives, as long as they provide the required variables for 'ferm' and 'firewalld' roles to consume. Initial deployment doesn't change from the current DebOps standards, but switching between different alternatives involves 3-step process. For example to switch from the default 'openntpd' service after it has been deployed, users have to: - run the playbook with 'ntp__deploy_state=absent' to remove openntpd and any dependent configuration such as that in the firewall - set the 'ntp__service=chrony' in the 'global-vars.yml' file. - run the playbook again, to apply the new service configuration Now, the thought in everyone's minds probably is "this is too complex". Yes, I agree with you all. But in actuality, the complexity stays the same, we are just moving it around and concentrating everything from "both sides": we merge the two playbooks together, and on the other side, we move the selection of what service to use from the service role into a separate 'wrapper role" which needs to be aware of all the possible choices to work correctly. Since all that complexity is now concentrated in one spot, that's why it looks to be bigger than it actually is. In fact, the final service roles, and the firewall roles, can be used just fine without the wrapper roles when we define a playbook for each combination separately. For example, 'ferm' firewall with 'chrony' NTP server: # ansible/playbooks/service/ferm-chrony.yml - name: Manage the chrony service with ferm firewall hosts: [ 'debops_service_chrony_ferm' ] become: True roles: - role: ferm tags: [ 'role::ferm', 'skip::ferm' ] ferm__dependent_rules: - '{{ chrony__ferm__dependent_rules }}' - role: chrony tags: [ 'role::chrony', 'skip::chrony' ] And we're back to the current state of the art. The actual implementation of all this would take some time. The 'ntp' role would have to be split, but we have most of the code written already so that's an easy step to take and could be done first as a proof of concept. A 'firewall' role could be easily created and only support 'ferm' for now, so that we can update all playbooks in anticipation for the new 'firewalld' and/or 'ufw' roles. For the webserver we could do the same, but the 'apache' role would have to be upgraded to support similar functionality to the 'nginx' role, not to mention that both probably should be rewritten from scratch with current configuration models in mind. But before I start messing around in the actual code, a question: is this overengineering? Am I going too far with this? All the trouble comes from the fact that DebOps roles are written with separation of concerns in mind, and pass the configuration for dependent roles via variables instead of directly messing with the services. It seems that this approach is not really common in the Ansible world outside of DebOps. Most of the time role creators write example configuration for a given role in the documentation and let the users deal with it on their own... But usually these roles focus on working with multiple Linux distributions. DebOps went essentially in an opposite direction, with focus on a single Linux distribution (Debian), but with the intention to provide as much support in that space as possible. An alternative approach would be to rip out the "glue" from the roles and playbooks and let the users deal with combining separate services together themselves. That would move the problem outside of Ansible in a way. Eventually we could deal with this via the rewritten 'debops' scripts which could manage the inventory for the users and combine different setups to generate playbooks for Ansible to use. This would let us deal with the complexity using a normal programming language (Python) instead of going with it through Ansible and its DSL. But we are a long way from that. Let me know what your thoughts are on the subject. -- Maciej [1]: https://lists.debops.org/hyperkitty/list/debops-devel@lists.debops.org/th... [2]: https://docs.debops.org/en/master/user-guide/project-directories.html#the...