On 13 Aug 2017, at 21:38 , Maciej Delmanowski <drybjed(a)drybjed.net&gt; wrote: Hello everyone, TL;DR: I want to merge all the Ansible roles, playbooks, documentation, tests and software repositories back into one DebOps git repository to improve development process. I want to be sure that it's done for the right reasons. A few days ago, Daniel Vetter published a blog post about how GitHub is not a good place to host the Linux kernel: http://blog.ffwll.ch/2017/08/github-why-cant-host-the-kernel.html In short, David explains how on GitHub big projects are commonly split into multiple git repositories, each one with their own issues and pull requests. Linux is developed using one "monotree", stored in a git repository that is forked multiple times, and changes to specific code are then merged back into the main Linux repository. While reading this blog post, I thought about how DebOps is currently maintained, what are the pain points and how the development process could be improved. But first... How did we end up here? ----------------------- DebOps wasn't always using multiple git repositories in one GitHub organization. In the beginning, the code with multiple roles and playbooks was in a single repository: first iteration: https://github.com/drybjed/ansible-aiua second iteration: https://github.com/drybjed/ginas/ There were two reasons at the time for a split into multiple git repositories, each repository containing one Ansible role: 1. Testing times on Travis-CI were approaching the 20-minute time limit, since the idea was to test as much Ansible roles as possible. With the number of roles increasing, there was a need to redesign the tests so that roles could be tested separately - multiple git repositories allowed for that. 2. There was a demand for some of the DebOps roles to be available via Ansible Galaxy, which supports a model where an Ansible role is in a separate git repository. In the end, all of the DebOps roles are available on Galaxy, but I'm not sure how many of them are used that way exclusively these days. So, the split happened and immediately it was apparent that in order for the project to work as expected, any role included in the playbook needed to be installed and available to Ansible. Since the 'debops-playbooks' repository, which at this point became a 'scaffolding" that joined the roles together, included all of the DebOps roles, all of them needed to be installed by the user. This automatically points to the usage of the Ansible Galaxy 'requirements.txt' file. However, DebOps project is a combination of Ansible roles that do the actual work, and Ansible playbooks which define what roles should be executed on which hosts. Ansible Galaxy only supports installation of roles, not playbooks, therefore there was a need for an automated way to download all of the project's git repositories and put them in the correct place for Ansible to use. And that is how the custom scripts came to be - at that point DebOps had 56 roles and that number was expected to increase, so there wasn't any better way to handle that otherwise. Over time, project evolved to the current state that it is today due to input from the users. The roles were tested using a separate test-suite repository which allowed to define a consistent test environment and decoupled any issues with tests from the git commits in role repositories themselves. Documentation went through a design phase where it was decided that due to the split nature of the git repositories, documentation for each role should be included with the respective role. All of them then were merged using 'git submodule' commands into one giant documentation repository and pushed to ReadTheDocs for consumption. The design of the role dependencies has changed from using "hard dependencies" in the roles themselves, to "soft dependencies" on the playbook level, so that roles could be used separately without the need to use the dependencies. Roles themselves became more and more self-contained, that led to design of the standardized ways the roles passed data around using role dependent variables and Ansible local facts. To ensure that the project's code is validated, git commits from project developers are now signed by their GPG keys, although code validation that would use this is not yet implemented. Due to how GitHub organization controls work, a separate 'debops-contrib' organization was created for third-party Ansible roles that are not yet part of the DebOps project, and are expected to be added at some point. What's the current state of the project --------------------------------------- In hindsight, some of these decisions were good (mostly related to the role code design and inter-role communication, at least in my opinion), and some were bad, but unavoidable within the selected development framework (using git submodules for documentation results in a very slow performance of the 'docs' repository). I think that the most glaring issue and the easiest to spot is the installation or update of all the git repositories that contain DebOps playbooks and roles. For comparsion, I cloned the Ansible repository (~139 MB) to a new directory and timed it: $ time git clone https://github.com/ansible/ansible 4,90s user 1,18s system 65% cpu 9,345 total Running the 'debops-update' script, which clones the 'debops-playbooks' repository and based on that, clones all of the DebOps role repositories (all of which has ~152 MB): $ time debops-update 2,20s user 0,81s system 2% cpu 2:13,13 total This is 10s for cloning 1 repository vs 2:13s for cloning 121 git repositories (updates of existing repositories are slightly faster). And this is usually done each time to get the latest changes, otherwise you would need to know what git repository changed and pull the changes manually. Some of the development process is required to be done by a human, notably GPG signing of each git commit or merge. Due to how GPG signing works, this cannot be done on GitHub itself through a web browser. When a pull request is accepted, the maintainer of a given role (in most cases, drybjed), fetches the involved branch from GitHub manually to a local git repository of a given role, merges and signs them, and pushes the new changes to GitHub. This cannot be changed or resolved without modifying the GPG signing process - using a bot to automatically sign commits requires a trusted infrastructure which the project doesn't have at the moment, and dropping the GPG signing may result in an untrusted code being introduced into the project. Since DebOps has essentially 'root' privileges in a production environment, not signing commits is not an option, especially that the code can be easily forked and hosted by third-parties (for example, https://github.com/rpmops/). We can't really do anything about this, but it's worth keeping in mind. Adding a new Ansible role to the existing infrastructure is an involved process in itself: - create new git repository on GitHub; - update list of known GitHub repositories on Travis-CI, enable testing of the new repository; - create new test in the 'test-suite' repository; - push the new role code to GitHub, check if tests on Travis-CI pass, fix any issues that arise in the test, or in the role itself; - when the new role passess successfully, tag it to mark a new release; - add the documentation of the new role to the 'docs' repository (very slow due to use of git submodules); - update the list of GitHub repositories in Ansible Galaxy, import the new role and ensure that it's correctly named; - add the role and its playbook to the 'debops-playbooks' repository, which officially enables the role in the DebOps project; After that, changes to existing DebOps roles are relatively easy to manage - after an upstream repository is forked, new commits are pushed there and a pull request is created on GitHub. When changes are accepted, the new changes are merged and signed locally. DebOps roles are versioned using git tags. This can be useful when you use a specific DebOps role in your own set of Ansible roles, but is mostly irrelevant for the 'debops-playbooks' repository which always pulls the latest commits in the 'master' branch of the DebOps roles, ignoring the tags. Currently role versioning matters most in the Travis-CI tests, which pull the required roles using Ansible Galaxy, which by default points to the latest tagged version of the role. Due to this, any changes that affect multiple roles (for example changes in 'debops.ferm' or recent changes to 'debops.postfix' need to be carefully coordinated, so that the main role is updated and tagged first, and then any roles that depend on it can be properly tested on Travis. The DebOps project currently does not have any concept of a "stable release". Individual roles are versioned, but the 'debops-playbooks' repository hasn't been tagged in a long time. The current project architecture doesn't point to any sane way to resolve this issue - there was an idea of creating separate git branches, each branch would contain the Ansible Galaxy 'requirements.yml' file with specific versions of each role, but that was quickly dropped due to being too much work to track everything manually (remember that each change would need to be GPG-signed). The current architecture of the project resulted in problems with packaging it for Debian: https://github.com/debops/docs/issues/132 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820367 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=819816 The documentation proved to be unsuitable for packkaging, therefore without any significant changes, I don't think that DebOps will be available in Debian Archive. The 'debops-contrib' GitHub organization hasn't been properly integrated into the project itself. The roles in this organization were meant to be moved to the main DebOps roganization after the current roles included in the project are updated to latest code standards. Unfortunately, this hasn't happended fast enough as I hoped, and I'm not sure when this will be picked up. It seems to me that the current development model results in greatly favoring the use of DebOps roles separately (they are properly versioned and taken care of, easily available through Ansible Galaxy), but this affects the usage of the project as a whole (no "stable" releases which makes the use of the project risky in production, any significant changes need to be carefully coordinated). Merging everything back together -------------------------------- Can the different DebOps roles, playbooks, documentation and other software git repositories be merged into one repository? With a bit of changes, yes. The current Ansible code within DebOps essentially is one "repository" split into separate git repositories - you can see it by looking at contents of the '~/.local/share/debops/' directory after installation. Merging the roles back with the playbooks shouldn't be a problem, apart from saving the git commit history. After the merge, some things could be moved around to improve the directory structure. Documentation of the roles could be moved to one 'docs/' directory, and cleaned up to remove redundant documentation like LICENSE files, etc. This would improve documentation management and allow to use links between different role documentation without the need to use separate link index files. This should also resolve the Debian packaging issue. Having one repository with roles and playbooks could enable easy creation of stable releases based on branches. The Semantic Versioning could be utilized to keep a few stable branches, while development is done in the 'master' branch. The repository could have reserved directories for custom roles or playbooks, which would allow the users to maintain their own fork and merge any changes in upstream with relative ease. Adding new roles to the project would be as easy as currently changing the existing ones without the overhead of creating new repositories, etc. Any changes that span multiple DebOps roles could be tracked in one pull request instead of separate ones for each role, and could be coordinated together. The ownership of the different roles or code could be managed by the CODEOWNERS/MAINTAINERS file which specifies which users should review any changes. There would be no need to maintain a separate 'debops-contrib' GitHub organization - third party roles could be prepared and merged in forked git repositories, or easily maintained in upstream repositories that apply changes from the DebOps repository. The use of the Travis-CI for tests is a problem in this model. Running the whole playbook at once takes too long, and some roles are mutually exclusive (for example 'apache' and 'nginx'), so using just one test for the whole project is not feasible. Travis-CI has a "build matrix" feature which allows to create separate jobs, which could be used to create a set of tests for different parts of the code, however the limit of maximum 200 jobs, and good behaviour suggests that this shouldn't be used to test all of the roles separately, as it is done now. Perhaps the main repository could do just a few tests for major roles and playbooks (ownCloud, GitLab, Netbox, ie. any user-facing application) which should test a relatively large part of the project codebase, and testing of all the roles separately could be done on a new infrastructure based on GitLab-CI. Some roles still might be popular enough to warrant their availability separately through Ansible Galaxy. This could be done automatically by extracting the selected roles and publishing them in their own separate git repositories, signed by a bot or a human, depending on number of roles. Ansible configuration allows to use multiple role directories using a 'roles_path' configuration variable with $PATH-like syntax, therefore cloning the main DebOps repository and adding the role directory in the 'roles_path' configuration variable shouldn't be a big issue. Roles would still be designed to be self-contained, so this use case will stay valid. The DebOps scripts will need to be updated to support the new deployment model. This might be a good enough reason to finally rewrite them from stratch and update the user interface to support subcommands. In similar fashion, third party code that was created to support DebOps could be merged in the main repository as well - example roles, test suite, example playbooks. We can carefully design the final directory structure to support this. Final thoughts -------------- This is just a proposal - I would like to hear the thoughts about this from other DebOps users before changing anything further. I think that proper way to do this would be first to review and update the DebOps Policy and Guidelines (https://github.com/debops/debops-policy/) to reflect the new state of the project. Then work can be done to merge all existing code into one repository to ensure that the commit history is preserved. After that, work can be done on redesigning the existing code, updating documentation, etc. Cheers, Maciej _______________________________________________ debops-users mailing list debops-users(a)lists.debops.org https://lists.debops.org/mailman/listinfo/debops-users

I'm not saying that all of the DebOps roles should be wiped from Ansible Galaxy completely. The current project model can be summarized as "create roles separately and then combine them in a bundle". What I'm proposing to do is reverse that into something like "create a consistent bundle of roles and expose the interesting ones on Ansible Galaxy".

In the curren state this "freeze" would need to be handled in the 'debops-playbooks' repository. Since the roles in separate git repositories, only mechanism that is valid to use to freeze on specific role versions is Ansible Galaxy 'requirements.yml' file, which would specify all of the roles and the desired versions.

Let's say that DebOps offers two "stable" branches, 1.x and 2.x, each one with a frozen set of roles. there's also separate 'master' branch which gets updated before being frozen. Now, a 'debops.nginx' role has some updates and a new version is released, now that version needs to be updated in the 'debops-playbooks' 'master' branch. And then 'debops.sshd', and 'debops.gitlab', etc., etc. Each release of a role requires a parallel update in 'debops-playbooks'. Note that right now an update in the playbook repository is needed only when a new role is added, or a playbook of an existing role is updated; otherwise 'debops-playbooks' is not changed all that much.

Without changing the development model of the project and implementing "freezes" like the above, that would put even more pressure on the project maintainers and add even more work. Can this be automated? Not without automating commit signing...

There was an attempt to do this by a Debian Developer a while ago (I linked the relevant Debian bugs in my initial post). Unfortunately this fell apart because of how the documentation is currently managed in DebOps via git submodules - this resulted in lots of duplicated role code in the 'debops-doc' package which couldn't be passed through Debian lintian. Keeping everything in one repository wouldn't have that issue and a 'debian/' directory with build instructions could be included as well.

On Aug 15, Reto Gantenbein wrote: As for the role being "interesting" enough to be published separately on Ansible Galaxy - this process will become highly automated, since the roles would not need to be tagged with GPG signatures (the original code will be signed and present in the main DebOps repository). If demand is high enough, all of the DebOps roles could be published that way on Galaxy using a bot. What will change is the direction the code is sourced from - instead of pulling all of the separate roles together into one directory structure, roles will be extracted from one "source of truth" git repository into separate satellite repositories.

> There are many other advantages as already nicely summarized by drybjed > in the initial mail, I don't have much to add here. Maybe one point > that I would like to mention that for new users/contributors it might > be much easier to grasp the idea of Ansible roles and DebOps if they > can easily read and run an individual role instead of a huge repository > with hundreds of roles and playbooks. At least I got this impression > when trying to dig into openshift-ansible [1] which is meant to setup a > OpenShift container PAAS and also consists of dozens of Ansible roles > with lots of possibilities and interpendencies. I don't think that the development model (one repository vs roles in separate repositories) changes this. At the end of the day, DebOps is yet another huge repository with hundreds of roles and playbooks, just laid out differently. New users still feel overwhelmed with the project, and need to learn it before trusting it. This is a documentation issue, and with a series of tutorials with encreasing levels of complexity should be relatively easy to overcome.

> Stability: > > DebOps definitely should have a way to "freeze" and properly test and > package a state of the Ansible roles which can consumed easily by > sysadmins who prefer to focus on application and architectural issues > and not bootstrap and configuration management issues. The current > distribution model of DebOps with updating the roles through git and > the constant rate of changes and redesigns of roles doesn't help to > build confidence. DebOps should help to make things easier for its > users not more complicated. In the curren state this "freeze" would need to be handled in the 'debops-playbooks' repository. Since the roles in separate git repositories, only mechanism that is valid to use to freeze on specific role versions is Ansible Galaxy 'requirements.yml' file, which would specify all of the roles and the desired versions. Let's say that DebOps offers two "stable" branches, 1.x and 2.x, each one with a frozen set of roles. there's also separate 'master' branch which gets updated before being frozen. Now, a 'debops.nginx' role has some updates and a new version is released, now that version needs to be updated in the 'debops-playbooks' 'master' branch. And then 'debops.sshd', and 'debops.gitlab', etc., etc. Each release of a role requires a parallel update in 'debops-playbooks'. Note that right now an update in the playbook repository is needed only when a new role is added, or a playbook of an existing role is updated; otherwise 'debops-playbooks' is not changed all that much. Without changing the development model of the project and implementing "freezes" like the above, that would put even more pressure on the project maintainers and add even more work. Can this be automated? Not without automating commit signing... The reason for this is that we are essentially duplicating the internal 'git' repository mechanisms using Ansible Galaxy 'requirements.yml' file. How would that change if everything is in a single repository? Any updates to a role are in the 'master' branch of the main git repository, there's no need to track role versions - in fact I'm not sure if role versioning would be needed at all in this case, at least not as frequent. The main git repository would be versioned as a whole, and exported DebOps roles on Ansible Galaxy could use the same version numbers. Export could be easily done by a bot, the roles published on Ansible Galaxy IMO don't need to be signed because a) they are already signed in the main DebOps repository, and b) 'ansible-galaxy' command downloads tarballs, not git repositories.

With one git repository there wouldn't be dozens of Changelogs to track, but only 1 Changelog. Stable releases would be required to not do any changes that break existing setups. Perhaps, with time, DebOps would also offer some way to migrate between different releases. Also, with much smaller number of git repositories to handle, detailed release notes are feasible to write.

> In a distributed world this CI machine would then also be responsible > for branching and tagging the individual repositories appropriately as > it is required to create releases and bugfix channels. Tracking all these repositories correctly quickly becomes a nightmare, and you are essentially reimplementing git to do this. Wouldn't it be better to use the existing git mechanisms (branches, tags) properly? > Consistency: > > There are already a nice bunch of policies and guides available for > DebOps but they have to be read and understood and there is no > automation available which checks and enforces those. With ansible-lint > [2] and ansible-review [3] there are already some tools available which > could be leveraged to improve and maintain consistency in this large > number of roles. That's a good idea to implement in the tests. +1.

> These were some of my thoughts. If some effort regarding packaging > should be made, I'm definitely willing to help as I'd love to see a > DebOps package in Debian (or at least on my servers). Thanks for your thoughts! I would also like to see DebOps as an official Debian package in a stable release. Unfortunately I don't think that is possible with the current way the project is maintained, so I'm willing to shake that up to see it happen.

> - The role maintainer model has to be given up (at least privilege > wise). Either one is a main developer of DebOps and has access to > everything or one has no rights. On the contrary, with just one repository, you can fork it on GitHub and gain access to everything that way. Your changes could then be pulled to the main repository from your fork. If your fork is popular enough that users start using it instead of the main DebOps repository, you win.

> - It's harder to follow development and see issues of individual roles > as everything will be mixed together. I think that this ties with the case where multiple roles need to be modified together to work properly (think project-wide firewall changes, or the recent Postfix updates). In the current model, each role requires a separate PR to handle the required changes, in one repository model all the needed changes could be grouped together in one PR, developed over time to encompass everything that's involved in that PR, and committed at once. Looking at issues for individual roles could be handled by properly using labels and/or tagging issue titles. In the current model I stopped using GitHub issue labels altogether because each repository needed to have them set up separately. I guess that this could be handled by writing scripts that manage labels in all DebOps repositories via GitHub API, but that never happened (lack of manpower, I suppose).

manage labels

On Aug 26, Robin Schneider wrote: >>> - The role maintainer model has to be given up (at least privilege >>> wise). Either one is a main developer of DebOps and has access to >>> everything or one has no rights. >> >> On the contrary, with just one repository, you can fork it on GitHub and gain >> access to everything that way. Your changes could then be pulled to the main >> repository from your fork. If your fork is popular enough that users start >> using it instead of the main DebOps repository, you win. > > I agree with ganto. Write privileges could only be given on a all or > nothing base. Some git receive hook should be put in place to limit this. > I was fine with having no write access to certain parts of the > project because I don’t/did not need them (principle of least privilege). I think that the "write access problem" is strictly linked to how the DebOps project is maintained on GitHub in an organization. With switch to one repository, when you clone it, you have access to the entire repository in your own fork. Just for comparsion of how Linux kernel is maintained this way - there's https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git git repository which is maintained by Linus, but there also hundreds of other git repositories with the kernel, maintained by different people, available at https://git.kernel.org/pub/scm/linux/kernel/. The one you "follow" depends on your choice. So, there could be a https://github.com/ypid/debops/ fork with ypid's changes, and https://github.com/ganto/debops/ fork with ganto's changes, and https://github.com/drybjed/debops/ fork with drybjed's changes. These forks are publicly available and users can choose to follow one of them, or cherrypick and combine stuff in their own fork. Now, what about https://github.com/debops/debops/ fork? With the current DebOps Policy in place, write access would be given to the current project Developers, ie. drybjed and ypid. The Developers (I imagine) would first put the changes in their own forks, and then they could be merged in the debops/debops tree after through testing. To be honest, with single git repository instead of multiple, the need for an entire separate 'debops/debops' fork kind of falls apart in my mind. Right now it should be used as a staging point to bring everything back together into a single repository. After that, when the new repository is forked by multiple people, we can see how it works out.

>>> - It's harder to follow development and see issues of individual roles >>> as everything will be mixed together. >> >> I think that this ties with the case where multiple roles need to be modified >> together to work properly (think project-wide firewall changes, or the recent >> Postfix updates). In the current model, each role requires a separate PR to >> handle the required changes, in one repository model all the needed changes >> could be grouped together in one PR, developed over time to encompass >> everything that's involved in that PR, and committed at once. >> >> Looking at issues for individual roles could be handled by properly using >> labels and/or tagging issue titles. In the current model I stopped using >> GitHub issue labels altogether because each repository needed to have them set >> up separately. I guess that this could be handled by writing scripts that >> manage labels in all DebOps repositories via GitHub API, but that never >> happened (lack of manpower, I suppose). > > There are both pros and cons here. I just want to emphasize one con for me: > > It would be much more difficult to watch only parts of DebOps. For example, I > maintain debops.cryptsetup and a few others and would need to watch for issues > and PRs. > Getting notified for all PRs/issues of DebOps directly does not scale. > > I guess the easiest solution would be a PR/issue template where contributors > need to CC the role maintainer. That should solve it. No need for that - https://help.github.com/articles/about-codeowners/. In short, it's a file in the root directory of the repository that specifies what files or paths are maintained by who, and when a new PR is published that changes one of these files, a person is added as a reviewer of that PR. Of course this works only on GitHub, but could be scripted elsewhere. Linux kernel uses a similar system with the MAINTAINERS file, which was an inspiration for the CODEOWNERS on GitHub.

> When the merge is done, I would like to do a review in that sense that nothing > of the role code was changed from the individual git repos (which I reviewed and > implicitly signed) to one repo and then create a signed commit. This way, the > merge which might be difficult for others to review is at least signed by two > independent parties. Scripts should be used for the merge where possible and it > would be nice if those scripts would also help with reviewing an already completed > merge. For now my plan is to resolve issues and pull requests in all roles and playbooks on GitHub. When there will be no more issues, I want to merge the roles and playbooks into one repository directly, preserving the commit history of each. The way it seems best to work is to first move all of the role files from the root of its repository to 'ansible/roles/' directory, then fetch the commits from the role repository to the destination repository and merge with 'master'. The same with the debops-playbooks repository and all the others. When everything is merged in one repository, and hopefully is still somewhat usable by Ansible, the result will be tagged as v0.9.0. After that, we can start working on merging all of the role documentation together, changelogs, and clean up. When it's all done, I hope that there will be a v1.0.0 release.

> I just saw that drybjed already initialized the new repo > (https://github.com/debops/debops). Now the rename of the debops tools to > debops/debops-tools pays off :) Yes, that was a good move. Right now I need to focus on finishing the Postfix redesign though, it's long overdue. Cheers, Maciej

On 13 Aug 2017, at 9:38 PM, Maciej Delmanowski <drybjed(a)drybjed.net&gt; wrote: Hello everyone, TL;DR: I want to merge all the Ansible roles, playbooks, documentation, tests and software repositories back into one DebOps git repository to improve development process. I want to be sure that it’s done for the right reasons.

What's the current state of the project --------------------------------------- In hindsight, some of these decisions were good (mostly related to the role code design and inter-role communication, at least in my opinion), and some were bad, but unavoidable within the selected development framework (using git submodules for documentation results in a very slow performance of the 'docs' repository). I think that the most glaring issue and the easiest to spot is the installation or update of all the git repositories that contain DebOps playbooks and roles. For comparsion, I cloned the Ansible repository (~139 MB) to a new directory and timed it: $ time git clone https://github.com/ansible/ansible 4,90s user 1,18s system 65% cpu 9,345 total Running the 'debops-update' script, which clones the 'debops-playbooks' repository and based on that, clones all of the DebOps role repositories (all of which has ~152 MB): $ time debops-update 2,20s user 0,81s system 2% cpu 2:13,13 total This is 10s for cloning 1 repository vs 2:13s for cloning 121 git repositories (updates of existing repositories are slightly faster). And this is usually done each time to get the latest changes, otherwise you would need to know what git repository changed and pull the changes manually.

> Debops in Debian will mean sticking with Debian release cycle. Roles will > have to work with the Debian release which includes them - freezed at > various versions. Some might stop working fast for example with non-freezed > webapps (e.g. owncloud). Unless they use owncloud from Debian packages > itself? Some roles might not be a good candidates for Debian packaging at > all. This would probably be better answered by interested Debian Developers, however I don't think that DebOps itself should strictly follow Debian release cycle.

recent improvements to Postfix support). Would you want to stick to the DebOps roles and playbooks that are included in Debian Buster, or would you rather use the more recent upstream version?

Down the line I think that there might be major DebOps releases that stick to Ansible versions supported by specific Debian release - that's the major dependency chain of the project. DebOps currently tries to support current stable Ansible release, and I'm not adding code that relies on Ansible features that hadn't been released in a stable release yet.

> (... multiple snips ...)

> It's not that important how the code is stored (separate/mono repo/tree) > - but how it's used (and tested). You can certainly test master branches of > all repos together - Travis might not be the best service to do it just. Using separate repositories becomes a problem when the maintenance is tied to a human via GPG signing. Let's say you want to introduce changes that affect multiple DebOps roles. In the current setup (multiple repositories, changes pushed via GitHub pull requests), each repository needs to be changed separately via its own pull request, each one signed separately by a human. In the monorepo, you can combine all of the changes (in separate commits) into one pull request against one repository. This is a big win for the human. The roles could still be split into separate repositories sourced from the monorepo; that version of the code could be signed by a bot with a GPG key that is signed by the DebOps Developers GPG keys. It's a bit weaker path of trust, but the code can be easily traced back to the monorepo.

> Debian often packages stuff using some basic packages and then - modules. > Checkout puppet-module-* packages for instance - very similar problem and > already in the tree. Still I'd consider keeping "core" playbooks together in > the basic package. From what I've seen from two attempts at packaging DebOps for Debian, packagers prefer merging the Ansible roles together in one package, but that might be due to the mentioned requirement of Ansible that all roles in a playbook need to be installed for the playbook to work.

> No idea why there can't be a tool to create a release tarball out of current > state of roles - including/excluding some and sticking to master branch - or > specific tags. Then do the packaging from this release. Treat the tarball as > "upstream" - not the GIT. There totally can be a tool that exports the code from the monorepo to a set of tarballs which are then published on some HTTP server. But tarballs cannot be the "upstream" because they lack proper version control. The "upstream" needs to be a proper source git repository.

So it seems that SHA1 cannot be reliably used to ensure that parent commits not signed by GPG are genuine. Combine this with the fact that at the moment DebOps is hosted and downloaded from GitHub. So yeah, I'm signing my commits and merges, manually. Of course this becomes a bottleneck with number of repositories, which switch to the monorepo will hopefully solve.

> (.. snips of things I'm unsure how to answer ..)

With a monorepo layout, everyone has access to their own fork of the DebOps repository and can create pull requests as they wish against other developers. Each DebOps Developer can have their own fork of the project and users can select which Developer they follow (compare that with Linux kernel where developers have their own git repositories on git.kernel.org and users can freely choose which developer they follow:

https://git.kernel.org/pub/scm/linux/kernel/). Since for Ansible Galaxy there would probably have to be one "official" set of roles, the debops/debops repository on GitHub could be shared between DebOps developers and two-way synchronization merges could be done against their own forks.

Then, a set of scripts could extract the roles from the debops/debops repository periodically, say on a new tagged release, and publish them on Ansible Galaxy.

The difference between the current and the new model is that instead of hundreds of source repositories combined into "one" virtual repository we have one source repository split into multiple ones. Much easier to handle by a human.

> (.. snip snip ..) > Not tying yourself tightly to github is certainly nice, but having multiple > repos doesn't necessary cause that. Multiple repositories are harder to handle without some synchronization tool like debops-update, which is currently tied to GitHub.