On Sep 07, Tomasz bla Fortuna wrote: I'll try explain the current workflows I use while working on DebOps, both when creating new roles and when updating existing ones. New role -------- When I create a new role from scratch, I start working on it in its new directory, usually with some stuff like docs/, .travis.yml, etc. copied from another role. I guess this could be better done with something like 'ansible-galaxy init', but I don't want to have any files/directories that I don't use. During development I'm starting to do git commits in about 2-3rd of the role being ready for a release, when it's general "shape" and functionality is clear enough. Each commit is signed. When a new role is ready, I go to GitHub and create a new repository for the role. Add a Travis-CI webhook, at the same time I go to travis-ci.org and update the list of repositories and enable the new ones. In the 'test-suite' repository, I create new subdirectory for the new role and create tests for it, roughly checking that during the test Ansible set everything up correctly. I push the 'test-suite' changes to GitHub. After that I push the new role to GitHub. The initial push is tested by Travis-CI. If everything is OK I tag the role with v0.1.0 release. If there are issues on Travis-CI, I solve them first before creating a new release. Sometimes issues are in the test environment itself (say, a wrong version of a package, or Ubuntu needs different role variables than the defaults). When the tests pass, I tag the new role. Currently the role releases are important for DebOps only on Travis-CI - the tests download the dependent roles using ansible-galaxy command, which picks the latest release of a given role. If there are modifications in a dependent role (say, debops.nginx) that are required by other roles, the dependent role needs to be tagged before tests on Travis-CI can pick the new changes up. After the new role is released, I go to Ansible Galaxy webpage, update the list of repositories and import the new role. After that it's availabla via 'ansible-galaxy' command. In my fork of the 'debops/docs' repository, I update all of the submodules to the current state (this repository is maintained by a bot that updates the documentation from all the roles) and add a new submodule with the new role. I push the changes to GitHub and if there are no issues detected by Travis-CI, I merge them with the upstream 'debops/docs' repository. This usually takes some time since the build needs to check out all of the submodules. When the documentation is up, in my fork of the 'debops/debops-playbooks' repository I add the playbook for the new role, with any dependencies it uses. The role is added to the 'galaxy/requirements.*' files via a script, these files are used by the 'debops-update' command to download or update all of the roles in the project. I push the changes to GitHub and if there are no issues on Travis-CI, I merge them to 'debops/debops-playbooks' repository. After this the new role will be noticed by the 'debops-update' script and installed. Update of an existing role -------------------------- I start by forking a 'debops/ansible-*' role to my user account on GitHub and cloning it to my development environment. I hook up the upstream repository as a 'git remote' for updates. I create a new git branch and work on an update. When the update is ready, I push the commits in the new branch to my fork (origin) on GitHub and create a new pull request against the upstream repository. All pull requests are tested on Travis-CI. I fix any issues with the update in my fork. When the tests pass, I go to the cloned upstream repository on my workstation, fetch the new PR and merge the changes - this way everything is signed by my GPG key. The buttons on GitHub are not used for merging. Merged changes are pushed to the repository on GitHub and if ready, a new release is tagged. I go to the Ansible Galaxy webpage and make sure that the new version is imported correctly. Similar process is done for any third-party pull requests; when the tests pass on Travis-CI, I fetch the changes and merge them manually so that the merge commit is signed. After that, 'debops-update' can pull new changes in roles automatically. The 'debops-playbooks' repository needs to be updated only if the changes modified the role playbook (for example, new dependent roles were added). The bot will update the 'debops/docs' repository automatically via git submodules. I've seen somewhere a comparsion to microservices. As I understand them, microservices are developed completely separately from each other and APIs are used to connect them together. DebOps could probably be considered a hybrid case of this. To understand this you need to know how role dependencies work together in DebOps. The DebOps roles and playbooks are designed to be read-only, that is you are not supposed to change them yourself, so that updates via 'git pull' can work as expected. To affect changes to roles to suit them to your environment, you can use Ansible inventory variables. All of the variables that the role exposes are loaded from the 'defaults/main.yml' file. This puts them at the "bottom" of the Ansible variable merge hierarchy, which means that any variables defined in Ansible inventory, or Ansible playbooks as dependent role variables, mask and override the ones from role defaults. To further allow modifications, variables that define the configuration of a dependent role are not stored directly in the playbook located in 'debops-playbooks'. Instead, they are also stored as variables in 'defaults/main.yml' of the role that uses the dependent role. Let's take an example to better illustrate it: The 'debops.mailman' role wants, among other things, to configure some options in the Postfix '/etc/postfix/main.cf' configuration file. But there's also a 'debops.postfix' role that manages the same file. If both of them are executed, the changes done by the first one will be overwritten by the second one. To combat this, the playbook of the 'debops.mailman' role has the 'debops.postfix' role as a dependency. This playbook entry defines 'postfix__dependent_maincf' variable with contents of the variable defined in the 'debops.mailman' role, 'defaults/main.yml' file. Ansible transparently uses this chained variable and the 'debops.postfix' role can configure the options specified by the 'debops.mailman' role on its behalf. The Postfix configuration is stored in the 'debops.mailman' default variables so that if the user wants to modify something, it can be done through Ansible inventory variables. If you don't want to use Postfix with Mailman because you prefer Exim, you can just remove the 'debops.postfix' entry from the 'debops.mailman' playbook (presumably your own copy). This makes maintenance of the "leaf" roles that are not used as dependencies very easy. However as recent Postfix update showed, the use of dependent role variables in other role defaults, in combination with DebOps tests on Travis-CI using 'ansible-galaxy' to download roles to preserve some kind of working order, is problematic when you want to do significant changes. When Postfix role rewrite happened, I needed to update all of the roles that used the 'debops.postfix' role as dependency (that is, 'debops.mailman', 'debops.dovecot', 'debops.smstools'), wait for the 'debops.postfix' changes to be merged in the upstream repository, merge the changes to the other roles, and update the 'debops-playbooks' repository to use the new playbooks. To summarize, DebOps roles are developed kind of like microservices since they can be used separately, but when more roles are involved so that the DebOps project as a while works as expected, they suddenly need to be developed together in a kind of lock-step fashion; otherwise they become desynchronized. As I understand it, this defeats the microservices methodology. In effect, DebOps roles are easy to work with an update separately, but larger changes involve more and more git repositories. This might not look that daunting from the user side - when all the changes are merged, one 'debops-update' run updates all of the involved git repositories in one run, but the development process becomes more and more involved since more and more roles in the project are using various dependencies. Since more and more roles use role dependencies, this makes the number of roles that are not needed by default to go down. I haven't looked, but perhaps 1/4th of the currnent roles could be not installed and the project would probably work fine without them. But I don't think that this number will increase, rather the role dependencies will be used more and more. DebOps is very tightly integrated at this point. The integration with external roles is currently done in the DebOps project directory. Custom roles and playbooks can be placed there and used alongside main DebOps playbooks. Switch to a monorepo wouldn't affect that mechanism. Yes, that would be the case with a monorepo. Another problem is that for the role to work correctly on Ansible Galaxy, files from the 'ansible/roles/<role-name>/' subdirectory in the monorepo would need to be moved to the root of the repository. The exported roles would also have a bilerplate README files that point to the monorepo, etc., so more commits that diverge the exported role from the monorepo. Since roles are meant to be read-only, the scripted flow of data from the monorepo to the separate role repositories would be one-way only. This of course hurts the contribution process for a single role, but might make it easier for a number of roles that depend on each other. This will affect the people that use DebOps roles separately more than the users of the whole project, ie. monorepo. The former would need to learn where to change the correct files in the monorepo and post the pull requests to the correct repository on GitHub. The question is, which user group is larger, and which one is more interested in contributions to the DebOps codebase. This can also be tied to the number of role maintainers - according to the data on the https://debops.org/status.html, there are two maintainers: - drybjed - maintains 74 roles - ypid - maintains 7 roles There are also 46 "unknown" roles which are not updated yet to be recognizable by the DebOps API. The maintainer of these roles will most likely be drybjed, unless somebody else interested steps in, so that is number of roles to maintain will be 120. That means that at the moment majority of the project upkeep relies on one maintainer. This leads to burnout. I'm not really sure how popular DebOps is. Because the project at least in my view is still in a beta state, I don't want to advertise it just yet because lots of things will still be changed. DebOps is focused on production environments, and implementing changes in production is tricky - most users that started using DebOps in production stopped updating the roles and playbooks because they feared that the implementation of the changes will break their production environments. Presumably having a way to reliably freeze the entire codebase in a particular state would help with that, but it's not realy feasible with the current mechanisms the project uses to keep the roles updated. Hopefully the change to monorepo would make the project easier to maintain in the long run, and after such a drastic change it would be ready to be more popularized, which would mean more potential developers. No no, keep it coming. I feel that the current state of DebOps development needs to change, and I would like to select a good course of action. From my perspective going with a monorepo seems to be a good choice, but other people might not see the whole picture, so I'm trying to explain it as best I can. If more people agree that going with a monorepo is a good choice, I will be sue that it's a correct one. With the explanation above I hope that you realize that currently most of these are done by one and the same person. ;-) I believe that's the case. Number of roles grows, so even if somehow we select a set of core roles in one repository with some in external repositories, after a while number of repositories will grow again and we're back to the same problem. Some of these were explained by me earlier, and some points could be already invalidated by my explanation on how the DebOps development process works. Let me know what you think about these after reading my explanation above. OK, but switch from one set of commits to another set of commits with significantly changing the number of signatures that need to be made by a human (presumably with most of the roles staying in separate repos) doesn't really change anything for that human. Any changes in the role repository would require a corresponding change on the list of know SHAs and therefore a commit as well. In some cases where in existing setup changes in the role does not require any changes in the 'debops-playbooks' repository, after your implementation number of required signed commits to be made would most likely increase as well. This would be great if most of the DebOps roles were maintained by other people. If these are maintained by me, signing a file in separate repository kind of misses the point. That changes only where the signed commits will go - instead of each role repository they would go to the global repository. I suppose that these are equivalent, since it amounts to creating a detached signature instead of an inline signature. It's easier to move one git repository between hosting providers than 120+ git repositories. Similarly it's easier to fork one git repository on GitHub than fork 20 git repositories because you need to make a related change to many of them to keep the whole thing synchronized. I know that this can be scripted and there are tools like mr to handle multiple repositories efficiently, but wouldn't it be better if the need for these wouldn't arise due to how the project itself is structured? Ansible Galaxy uses GitHub as storage, so to publish roles they would need to be in a separate GitHub repository. This basically equates to "hunders of tarballs". Most of the DebOps roles were written by me, just a few of them were started by other people. Hopefully more through, automated testing of roles will make finding out bitrot easier and help with maintenance. With monorepo the export would be automated and one way only, development would be focused on monorepo. I imagine that with use of the API, the scripts could even handle creation of new GitHub repositories and registration in Ansible Galaxy. You can check the download stats using this link: https://galaxy.ansible.com/list#/roles?page=1&page_size=20&users=... It looks like ~22 roles have >= 1000 downloads, roughly ~30 roles have fewer than 100 downloads. Remember that DebOps roles weren't cherrypicked to be published on Galaxy and these results came over time. Since Ansible Galaxy is currently tied to testing on Travis-CI, all project roles need to be available on Galaxy. When project switches to an internal testing infrastructure, that requirement will be dropped. Since the process to publish DebOps roles on Galaxy will be automated, I don't see a reason not to publish all of them. That would be excellent. :-) I would wait a bit though. One, not all git repositories are signed, and two, I'm not sure yet how the change in development will affect the deployment of roles and playbooks in ~/.local/share/debops/. Cheers, Maciej