Re: [debops-users] RFC: Merge all DebOps git repositories into one
by Maciej Delmanowski
On Sep 11, Tomasz bla Fortuna wrote:
>> I'll try explain the current workflows I use while working on DebOps, both
>> when creating new roles and when updating existing ones.
> Very good description for the docs indeed. ;)
I suppose, but this will change, and would need to be written more from
a contributor perspective. Will do, eventually. :)
> Summarizing from my point of view:
> - You are, along ypid, pretty much the only persistent devs in this
> project.
> - And you both agreed that merging will help you work with it.
> (as long as notifications / templates are done well)
> - If it helps you - most other users should be pragmatically happy
> about it too.
>
> So I believe you should go with it.
Great. I'm already resolving old PRs (32 left) and issues (288 left) in the
GitHub repositories.
> Things I would consider:
> - The roles that need to work separately should still be tested
> separately after the merge to maintain that status.
> Dependencies like to creep in.
> Currently that's the only way the tests are done, but that is a
> subject to change eventually I guess.
Of course. Each role will still be tested separately as well as in interesting
groups. In fact, the GitLab test pipeline will have a few steps - basic
Ansible syntax check + documentation check, then some base DebOps roles, then
all of the rest of the roles, then the main playbook. Hopefully the order will
be that so the easy to spot issues will be found out before heavy tests.
> - I believe the tooling should be able to use external roles, not
> included in the main repo with ease. That way you can handle
> maintainers of external roles who would:
>
> - Use debops normally,
> - develop locally their own role,
> - publish in on github and letting it stabilize before including
> directly in debops. (such role might be even alternative approach
> to postfix which would never get included in fact)
> - it would allow a user to include a number of external roles easily
> and keep them updated.
>
> (Possibly some glue code would have to be generated (including not
> only all.yml but also external.yml which would have to be left out of
> git but generated after clone? According to docs in 2.4 include is
> deprecated and there are some ways to do dynamic includes)
The glue code is basically the 'debops' scripts were meant to be, and this of
course stays. I'll need to look at that dynamic includes, might be worth it.
Cheers,
Maciej
6 years, 7 months
- 1
- 0
Re: [debops-users] RFC: Merge all DebOps git repositories into one
by Maciej Delmanowski
On Sep 07, Tomasz bla Fortuna wrote:
> - Doesn't increase complexity - unless I don't understand the current
> workflow well enough. If it does - I don't see how. I'm trying to
> explain below.
I'll try explain the current workflows I use while working on DebOps, both
when creating new roles and when updating existing ones.
New role
--------
When I create a new role from scratch, I start working on it in its new
directory, usually with some stuff like docs/, .travis.yml, etc. copied from
another role. I guess this could be better done with something like
'ansible-galaxy init', but I don't want to have any files/directories that
I don't use. During development I'm starting to do git commits in about 2-3rd
of the role being ready for a release, when it's general "shape" and
functionality is clear enough. Each commit is signed.
When a new role is ready, I go to GitHub and create a new repository for the
role. Add a Travis-CI webhook, at the same time I go to travis-ci.org and
update the list of repositories and enable the new ones. In the 'test-suite'
repository, I create new subdirectory for the new role and create tests for
it, roughly checking that during the test Ansible set everything up correctly.
I push the 'test-suite' changes to GitHub.
After that I push the new role to GitHub. The initial push is tested by
Travis-CI. If everything is OK I tag the role with v0.1.0 release. If there
are issues on Travis-CI, I solve them first before creating a new release.
Sometimes issues are in the test environment itself (say, a wrong version of
a package, or Ubuntu needs different role variables than the defaults). When
the tests pass, I tag the new role.
Currently the role releases are important for DebOps only on Travis-CI - the
tests download the dependent roles using ansible-galaxy command, which picks
the latest release of a given role. If there are modifications in a dependent
role (say, debops.nginx) that are required by other roles, the dependent role
needs to be tagged before tests on Travis-CI can pick the new changes up.
After the new role is released, I go to Ansible Galaxy webpage, update the
list of repositories and import the new role. After that it's availabla via
'ansible-galaxy' command.
In my fork of the 'debops/docs' repository, I update all of the submodules to
the current state (this repository is maintained by a bot that updates the
documentation from all the roles) and add a new submodule with the new role.
I push the changes to GitHub and if there are no issues detected by Travis-CI,
I merge them with the upstream 'debops/docs' repository. This usually takes
some time since the build needs to check out all of the submodules.
When the documentation is up, in my fork of the 'debops/debops-playbooks'
repository I add the playbook for the new role, with any dependencies it uses.
The role is added to the 'galaxy/requirements.*' files via a script, these
files are used by the 'debops-update' command to download or update all of the
roles in the project. I push the changes to GitHub and if there are no issues
on Travis-CI, I merge them to 'debops/debops-playbooks' repository. After this
the new role will be noticed by the 'debops-update' script and installed.
Update of an existing role
--------------------------
I start by forking a 'debops/ansible-*' role to my user account on GitHub and
cloning it to my development environment. I hook up the upstream repository as
a 'git remote' for updates. I create a new git branch and work on an update.
When the update is ready, I push the commits in the new branch to my fork
(origin) on GitHub and create a new pull request against the upstream
repository. All pull requests are tested on Travis-CI. I fix any issues with
the update in my fork. When the tests pass, I go to the cloned upstream
repository on my workstation, fetch the new PR and merge the changes - this
way everything is signed by my GPG key. The buttons on GitHub are not used for
merging. Merged changes are pushed to the repository on GitHub and if ready,
a new release is tagged. I go to the Ansible Galaxy webpage and make sure that
the new version is imported correctly.
Similar process is done for any third-party pull requests; when the tests pass
on Travis-CI, I fetch the changes and merge them manually so that the merge
commit is signed. After that, 'debops-update' can pull new changes in roles
automatically.
The 'debops-playbooks' repository needs to be updated only if the changes
modified the role playbook (for example, new dependent roles were added). The
bot will update the 'debops/docs' repository automatically via git submodules.
> - It doesn't simplify things *as much* as monorepo does, but this can
> be traded depending how much packages needs to live on Galaxy.
> - Has a some advantages over monorepo:
> * Roles working separately without dependencies on the rest of debops
> can be developed separately without synthetic solutions which might
> cause problems.
>
> * It's generally easier to combine software together, then to split
> software. From my experience (with working with bad developers
> probably! :D) without a clear boundaries the dependencies between
> modules tend to grow - like entropy.
I've seen somewhere a comparsion to microservices. As I understand them,
microservices are developed completely separately from each other and APIs are
used to connect them together. DebOps could probably be considered a hybrid
case of this. To understand this you need to know how role dependencies work
together in DebOps.
The DebOps roles and playbooks are designed to be read-only, that is you are
not supposed to change them yourself, so that updates via 'git pull' can work
as expected. To affect changes to roles to suit them to your environment, you
can use Ansible inventory variables. All of the variables that the role
exposes are loaded from the 'defaults/main.yml' file. This puts them at the
"bottom" of the Ansible variable merge hierarchy, which means that any
variables defined in Ansible inventory, or Ansible playbooks as dependent role
variables, mask and override the ones from role defaults.
To further allow modifications, variables that define the configuration of
a dependent role are not stored directly in the playbook located in
'debops-playbooks'. Instead, they are also stored as variables in
'defaults/main.yml' of the role that uses the dependent role. Let's take an
example to better illustrate it:
The 'debops.mailman' role wants, among other things, to configure some options
in the Postfix '/etc/postfix/main.cf' configuration file. But there's also
a 'debops.postfix' role that manages the same file. If both of them are
executed, the changes done by the first one will be overwritten by the second
one. To combat this, the playbook of the 'debops.mailman' role has the
'debops.postfix' role as a dependency. This playbook entry defines
'postfix__dependent_maincf' variable with contents of the variable defined in
the 'debops.mailman' role, 'defaults/main.yml' file. Ansible transparently
uses this chained variable and the 'debops.postfix' role can configure the
options specified by the 'debops.mailman' role on its behalf. The
Postfix configuration is stored in the 'debops.mailman' default variables so
that if the user wants to modify something, it can be done through Ansible
inventory variables. If you don't want to use Postfix with Mailman because you
prefer Exim, you can just remove the 'debops.postfix' entry from the
'debops.mailman' playbook (presumably your own copy).
This makes maintenance of the "leaf" roles that are not used as dependencies
very easy. However as recent Postfix update showed, the use of dependent role
variables in other role defaults, in combination with DebOps tests on
Travis-CI using 'ansible-galaxy' to download roles to preserve some kind of
working order, is problematic when you want to do significant changes. When
Postfix role rewrite happened, I needed to update all of the roles that used
the 'debops.postfix' role as dependency (that is, 'debops.mailman',
'debops.dovecot', 'debops.smstools'), wait for the 'debops.postfix' changes to
be merged in the upstream repository, merge the changes to the other roles,
and update the 'debops-playbooks' repository to use the new playbooks.
To summarize, DebOps roles are developed kind of like microservices since they
can be used separately, but when more roles are involved so that the DebOps
project as a while works as expected, they suddenly need to be developed
together in a kind of lock-step fashion; otherwise they become desynchronized.
As I understand it, this defeats the microservices methodology. In effect,
DebOps roles are easy to work with an update separately, but larger changes
involve more and more git repositories. This might not look that daunting from
the user side - when all the changes are merged, one 'debops-update' run
updates all of the involved git repositories in one run, but the development
process becomes more and more involved since more and more roles in the
project are using various dependencies.
> * Can in a natural way combine build-in/core roles with "external"
> roles while keeping both secure. This is less useful if all roles
> must be always available during the run. I believe that for
> packaging roles for Debian it would be better to allow some roles to be
> removed during the "build process".
Since more and more roles use role dependencies, this makes the number of
roles that are not needed by default to go down. I haven't looked, but perhaps
1/4th of the currnent roles could be not installed and the project would
probably work fine without them. But I don't think that this number will
increase, rather the role dependencies will be used more and more. DebOps is
very tightly integrated at this point.
> * Monorepo/tree works for Linux certainly but they don't care if some
> stuff is used outside of Linux.
The integration with external roles is currently done in the DebOps project
directory. Custom roles and playbooks can be placed there and used alongside
main DebOps playbooks. Switch to a monorepo wouldn't affect that mechanism.
> * I wonder if exporting Galaxy roles won't cause problems like:
> someone wants to fix a role, but he is using Galaxy. You will get a
> pull-request (and forks) to the exported "read-only" GIT repo. The
> valid way would be to fix the main repo, test it and export the
> role back.
Yes, that would be the case with a monorepo. Another problem is that for the
role to work correctly on Ansible Galaxy, files from the
'ansible/roles/<role-name>/' subdirectory in the monorepo would need to be
moved to the root of the repository. The exported roles would also have
a bilerplate README files that point to the monorepo, etc., so more commits
that diverge the exported role from the monorepo. Since roles are meant to be
read-only, the scripted flow of data from the monorepo to the separate role
repositories would be one-way only.
This of course hurts the contribution process for a single role, but might
make it easier for a number of roles that depend on each other. This will
affect the people that use DebOps roles separately more than the users of the
whole project, ie. monorepo. The former would need to learn where to change
the correct files in the monorepo and post the pull requests to the correct
repository on GitHub.
The question is, which user group is larger, and which one is more interested
in contributions to the DebOps codebase.
This can also be tied to the number of role maintainers - according to the
data on the https://debops.org/status.html, there are two maintainers:
- drybjed - maintains 74 roles
- ypid - maintains 7 roles
There are also 46 "unknown" roles which are not updated yet to be recognizable
by the DebOps API. The maintainer of these roles will most likely be drybjed,
unless somebody else interested steps in, so that is number of roles to
maintain will be 120. That means that at the moment majority of the project
upkeep relies on one maintainer. This leads to burnout.
I'm not really sure how popular DebOps is. Because the project at least in my
view is still in a beta state, I don't want to advertise it just yet because
lots of things will still be changed. DebOps is focused on production
environments, and implementing changes in production is tricky - most users
that started using DebOps in production stopped updating the roles and
playbooks because they feared that the implementation of the changes will
break their production environments. Presumably having a way to reliably
freeze the entire codebase in a particular state would help with that, but
it's not realy feasible with the current mechanisms the project uses to keep
the roles updated. Hopefully the change to monorepo would make the project
easier to maintain in the long run, and after such a drastic change it would
be ready to be more popularized, which would mean more potential developers.
> With two options (both easier and better then the current state - I
> believe) you can make a decision. Tell me if you don't see any
> advantages here though and I'll cease my blabbering. ;)
No no, keep it coming. I feel that the current state of DebOps development
needs to change, and I would like to select a good course of action. From my
perspective going with a monorepo seems to be a good choice, but other people
might not see the whole picture, so I'm trying to explain it as best I can. If
more people agree that going with a monorepo is a good choice, I will be sue
that it's a correct one.
> I'll answer some misunderstandings in detail below, but in short:
>
> You would end up with following project roles:
> - DebOps maintainer (signs, verifies, updates verified commit list in a main repo)
> - Core role developer (manages roles stored in the main repo)
> - Role developer (manages a role held in a separate repo, has contact
> with maintainer is responsible to keep the role compatible with current
> reviewed debops version).
With the explanation above I hope that you realize that currently most of
these are done by one and the same person. ;-)
> In the perfect world you would share debops maintainer + core role
> developer roles and work with a single repo only. I believe you'd also have
> to maintain some external roles (but I lack knowledge how many. And
> hence I cannot predict the feasibility of this approach).
>
> If in this split you end up managing over 10 repos (after merging core
> roles into main repo) then it doesn't solve your problem and you have
> no alternative to monorepo really.
I believe that's the case. Number of roles grows, so even if somehow we select
a set of core roles in one repository with some in external repositories,
after a while number of repositories will grow again and we're back to the
same problem.
>> If I understand this correctly, you propose to use the current design of
>> multiple Ansible role repositories + debops-playbooks repository, and add yet
>> another git repository separate from these that tracks everything by the SHA1
>> hashes.
> I'd use existing repository - debops playbooks. Just add a signed
> information about "stable/checked" commit SHA-1 of all "slave"
> repositories - in lieu of signing commits.
>
> Certainly not a "new" repo - I'm trying to limit parts you have to
> watch too.
>
> Also - any roles considered "core" could be still migrated to this repo
> to further limit moving parts:
>
> - Required, "core" roles managed by the core team can be held in
> the same repo. Which reduces the number of things to track for the
> maintainer.
> - Still other roles:
> * less stable,
> * the ones which can work by themselves, used on galaxy
> * maintained by other people with pull-requests (PR)
> can be held separately and still be used to securely "build" a debops
> distribution.
>
> I regret that I don't know currently how many roles classify to each
> group. That can lower the feasibility of this solution. ;)
>
> In monorepo case having to export some of the roles to external
> repositories to make them usable in Galaxy increases the moving parts a
> bit too and is a bit weird. Certainly doable though.
Some of these were explained by me earlier, and some points could be already
invalidated by my explanation on how the DebOps development process works. Let
me know what you think about these after reading my explanation above.
>> This failed
>> spectacularly because it just added more stuff to track, namely what Ansible
>> role versions worked with other role versions.
> I'm at most trying to substitute things you're tracking now (commits in
> multiple repos which needs to be signed with pull-requests etc.) with
> other things (list of SHA-s of accepted commits).
>
> I'm not trying to change the way the code works now. That is
> "everything works together" is imho ok - without versions between roles
> and dependencies.
OK, but switch from one set of commits to another set of commits with
significantly changing the number of signatures that need to be made by
a human (presumably with most of the roles staying in separate repos) doesn't
really change anything for that human. Any changes in the role repository
would require a corresponding change on the list of know SHAs and therefore
a commit as well. In some cases where in existing setup changes in the role
does not require any changes in the 'debops-playbooks' repository, after your
implementation number of required signed commits to be made would most likely
increase as well.
>> I'm sorry, but I like writing Ansible roles and design what they do, not
>> micromanage software versions. Remember that the number of DebOps roles almost
>> always grows, and almost never shrinks. Managing Ansible roles separately
>> without a dedicated team will become harder and more time-consuming as the
>> project grows.
> I understand. Assuming you currently go over any new code in each repo
> and tag+sign it - I'm not proposing anything new. Just leaving out:
> - Separate signing of each repo (sign one file after a review instead)
>
> - doing pull-request to repos you don't manage.
> And make the maintainers of other repos inform you that there's
> something worth looking at by doing a Pull-request to your repo (which
> is the only one a maintainer would have to track.)
>
> This of course splits code a bit into "reviewed/stable" part and
> "not-yet-reviewed" part - but I understand that's the case currently
> too?
This would be great if most of the DebOps roles were maintained by other
people. If these are maintained by me, signing a file in separate repository
kind of misses the point.
>> How the 'verified state' is maintained?
> Currently you sign commits in repos (but then debops-update uses master
> anyway) if I understand correctly. I just suggest signing a file
> pointing to repos instead. Without adding any more versioning and
> dependencies.
That changes only where the signed commits will go - instead of each role
repository they would go to the global repository. I suppose that these are
equivalent, since it amounts to creating a detached signature instead of an
inline signature.
>> I don't think that can be automated,
>> since the necessary information about what role versions work with other role
>> versions isn't there.
> For a given commit of master debops-playbook repos there would always
> be listed commits in other repos which were reviewed and used at that
> time - pretty much exactly like with monorepo, but allowing them to
> reside in seperate git repos.
It's easier to move one git repository between hosting providers than 120+ git
repositories. Similarly it's easier to fork one git repository on GitHub than
fork 20 git repositories because you need to make a related change to many of
them to keep the whole thing synchronized. I know that this can be scripted
and there are tools like mr to handle multiple repositories efficiently, but
wouldn't it be better if the need for these wouldn't arise due to how the
project itself is structured?
>> I propose to curb that into:
>> one git repo -> hundreds of tarballs
> I don't really understand why multiple tarballs though. Single tarball
> is fine, single debian source package is fine - this one can generate
> few debian packages (debops, debops-docs at least).
Ansible Galaxy uses GitHub as storage, so to publish roles they would need to
be in a separate GitHub repository. This basically equates to "hunders of
tarballs".
>> The thing is, people change,
>> people get bored, people remove the repositories they are not interested in
>> maintaining anymore. I predict that this model would quickly lead to parts of
>> code used by 'debops' scripts (since it wouldn't be in the main DebOps project
>> anymore) suddenly missing.
> True. And you always need to make a decision then:
> - Do I have the time and resources to maintain this role myself? Do I
> keep it in even if the code will deteriorate in time?
Most of the DebOps roles were written by me, just a few of them were started
by other people. Hopefully more through, automated testing of roles will make
finding out bitrot easier and help with maintenance.
>> And don't forget that I still need to worry about my huge set of git
>> repositories with DebOps roles that I maintain. Your solution wouldn't help
>> with that, just add more on top of it.
> With roles that doesn't need to go on Galaxy you could include them in
> the main repo. Those working in Galaxy need a separate repo and
> additional maintenance also in monorepo case.
With monorepo the export would be automated and one way only, development
would be focused on monorepo. I imagine that with use of the API, the scripts
could even handle creation of new GitHub repositories and registration in
Ansible Galaxy.
>> (snip, snip) everything back. DebOps has ~120 roles, that's ~120
>> submodules. This doesn't scale.
> How many roles could be considered core and how many should work
> separately on Galaxy? Is it few 5-20 roles for galaxy or rather most of
> 120 roles?
You can check the download stats using this link:
https://galaxy.ansible.com/list#/roles?page=1&page_size=20&users=debops&o...
It looks like ~22 roles have >= 1000 downloads, roughly ~30 roles have fewer
than 100 downloads. Remember that DebOps roles weren't cherrypicked to be
published on Galaxy and these results came over time.
Since Ansible Galaxy is currently tied to testing on Travis-CI, all project
roles need to be available on Galaxy. When project switches to an internal
testing infrastructure, that requirement will be dropped. Since the process to
publish DebOps roles on Galaxy will be automated, I don't see a reason not to
publish all of them.
>> Yes. I'm more focused on Ansible roles than on the 'debops' scripts, for me
>> they are "just enough" but I imagine that for other users this might be
>> different. Having a proper scripts that validate that the role repositories
>> are correctly signed would be useful, but it's not enough if an itch for me
>> personally to work on it.
> I'm a Python developer so if you believe this might be useful I can
> implement it in an agreed fashion.
That would be excellent. :-) I would wait a bit though. One, not all git
repositories are signed, and two, I'm not sure yet how the change in
development will affect the deployment of roles and playbooks in
~/.local/share/debops/.
Cheers,
Maciej
6 years, 7 months
- 2
- 1
[debops-users] RFC: Merge all DebOps git repositories into one
by Maciej Delmanowski
Hello everyone,
TL;DR: I want to merge all the Ansible roles, playbooks, documentation, tests
and software repositories back into one DebOps git repository to improve
development process. I want to be sure that it's done for the right reasons.
A few days ago, Daniel Vetter published a blog post about how GitHub is not
a good place to host the Linux kernel:
http://blog.ffwll.ch/2017/08/github-why-cant-host-the-kernel.html
In short, David explains how on GitHub big projects are commonly split into
multiple git repositories, each one with their own issues and pull requests.
Linux is developed using one "monotree", stored in a git repository that is
forked multiple times, and changes to specific code are then merged back into
the main Linux repository.
While reading this blog post, I thought about how DebOps is currently
maintained, what are the pain points and how the development process could be
improved. But first...
How did we end up here?
-----------------------
DebOps wasn't always using multiple git repositories in one GitHub
organization. In the beginning, the code with multiple roles and playbooks was
in a single repository:
first iteration: https://github.com/drybjed/ansible-aiua
second iteration: https://github.com/drybjed/ginas/
There were two reasons at the time for a split into multiple git repositories,
each repository containing one Ansible role:
1. Testing times on Travis-CI were approaching the 20-minute time limit, since
the idea was to test as much Ansible roles as possible. With the number of
roles increasing, there was a need to redesign the tests so that roles could
be tested separately - multiple git repositories allowed for that.
2. There was a demand for some of the DebOps roles to be available via Ansible
Galaxy, which supports a model where an Ansible role is in a separate git
repository. In the end, all of the DebOps roles are available on Galaxy, but
I'm not sure how many of them are used that way exclusively these days.
So, the split happened and immediately it was apparent that in order for the
project to work as expected, any role included in the playbook needed to be
installed and available to Ansible. Since the 'debops-playbooks' repository,
which at this point became a 'scaffolding" that joined the roles together,
included all of the DebOps roles, all of them needed to be installed by the
user.
This automatically points to the usage of the Ansible Galaxy 'requirements.txt'
file. However, DebOps project is a combination of Ansible roles that do the
actual work, and Ansible playbooks which define what roles should be executed
on which hosts. Ansible Galaxy only supports installation of roles, not
playbooks, therefore there was a need for an automated way to download all of
the project's git repositories and put them in the correct place for Ansible to
use. And that is how the custom scripts came to be - at that point DebOps had
56 roles and that number was expected to increase, so there wasn't any better
way to handle that otherwise.
Over time, project evolved to the current state that it is today due to input
from the users. The roles were tested using a separate test-suite repository
which allowed to define a consistent test environment and decoupled any issues
with tests from the git commits in role repositories themselves. Documentation
went through a design phase where it was decided that due to the split nature
of the git repositories, documentation for each role should be included with
the respective role. All of them then were merged using 'git submodule'
commands into one giant documentation repository and pushed to ReadTheDocs for
consumption. The design of the role dependencies has changed from using "hard
dependencies" in the roles themselves, to "soft dependencies" on the playbook
level, so that roles could be used separately without the need to use the
dependencies. Roles themselves became more and more self-contained, that led to
design of the standardized ways the roles passed data around using role
dependent variables and Ansible local facts. To ensure that the project's code
is validated, git commits from project developers are now signed by their GPG
keys, although code validation that would use this is not yet implemented. Due
to how GitHub organization controls work, a separate 'debops-contrib'
organization was created for third-party Ansible roles that are not yet part of
the DebOps project, and are expected to be added at some point.
What's the current state of the project
---------------------------------------
In hindsight, some of these decisions were good (mostly related to the role
code design and inter-role communication, at least in my opinion), and some
were bad, but unavoidable within the selected development framework (using git
submodules for documentation results in a very slow performance of the 'docs'
repository). I think that the most glaring issue and the easiest to spot is the
installation or update of all the git repositories that contain DebOps
playbooks and roles. For comparsion, I cloned the Ansible repository (~139 MB)
to a new directory and timed it:
$ time git clone https://github.com/ansible/ansible
4,90s user 1,18s system 65% cpu 9,345 total
Running the 'debops-update' script, which clones the 'debops-playbooks'
repository and based on that, clones all of the DebOps role repositories (all
of which has ~152 MB):
$ time debops-update
2,20s user 0,81s system 2% cpu 2:13,13 total
This is 10s for cloning 1 repository vs 2:13s for cloning 121 git repositories
(updates of existing repositories are slightly faster). And this is usually
done each time to get the latest changes, otherwise you would need to know what
git repository changed and pull the changes manually.
Some of the development process is required to be done by a human, notably
GPG signing of each git commit or merge. Due to how GPG signing works, this
cannot be done on GitHub itself through a web browser. When a pull request is
accepted, the maintainer of a given role (in most cases, drybjed), fetches the
involved branch from GitHub manually to a local git repository of a given role,
merges and signs them, and pushes the new changes to GitHub. This cannot be
changed or resolved without modifying the GPG signing process - using a bot to
automatically sign commits requires a trusted infrastructure which the project
doesn't have at the moment, and dropping the GPG signing may result in an
untrusted code being introduced into the project. Since DebOps has essentially
'root' privileges in a production environment, not signing commits is not an
option, especially that the code can be easily forked and hosted by
third-parties (for example, https://github.com/rpmops/). We can't really do
anything about this, but it's worth keeping in mind.
Adding a new Ansible role to the existing infrastructure is an involved process
in itself:
- create new git repository on GitHub;
- update list of known GitHub repositories on Travis-CI, enable testing of the
new repository;
- create new test in the 'test-suite' repository;
- push the new role code to GitHub, check if tests on Travis-CI pass, fix any
issues that arise in the test, or in the role itself;
- when the new role passess successfully, tag it to mark a new release;
- add the documentation of the new role to the 'docs' repository (very slow due
to use of git submodules);
- update the list of GitHub repositories in Ansible Galaxy, import the new role
and ensure that it's correctly named;
- add the role and its playbook to the 'debops-playbooks' repository, which
officially enables the role in the DebOps project;
After that, changes to existing DebOps roles are relatively easy to manage
- after an upstream repository is forked, new commits are pushed there and
a pull request is created on GitHub. When changes are accepted, the new changes
are merged and signed locally.
DebOps roles are versioned using git tags. This can be useful when you use
a specific DebOps role in your own set of Ansible roles, but is mostly
irrelevant for the 'debops-playbooks' repository which always pulls the latest
commits in the 'master' branch of the DebOps roles, ignoring the tags.
Currently role versioning matters most in the Travis-CI tests, which pull the
required roles using Ansible Galaxy, which by default points to the latest
tagged version of the role. Due to this, any changes that affect multiple roles
(for example changes in 'debops.ferm' or recent changes to 'debops.postfix'
need to be carefully coordinated, so that the main role is updated and tagged
first, and then any roles that depend on it can be properly tested on Travis.
The DebOps project currently does not have any concept of a "stable release".
Individual roles are versioned, but the 'debops-playbooks' repository hasn't
been tagged in a long time. The current project architecture doesn't point to
any sane way to resolve this issue - there was an idea of creating separate git
branches, each branch would contain the Ansible Galaxy 'requirements.yml' file
with specific versions of each role, but that was quickly dropped due to being
too much work to track everything manually (remember that each change would
need to be GPG-signed).
The current architecture of the project resulted in problems with packaging it
for Debian:
https://github.com/debops/docs/issues/132
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820367
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=819816
The documentation proved to be unsuitable for packkaging, therefore without any
significant changes, I don't think that DebOps will be available in Debian
Archive.
The 'debops-contrib' GitHub organization hasn't been properly integrated into
the project itself. The roles in this organization were meant to be moved to
the main DebOps roganization after the current roles included in the project
are updated to latest code standards. Unfortunately, this hasn't happended fast
enough as I hoped, and I'm not sure when this will be picked up.
It seems to me that the current development model results in greatly favoring
the use of DebOps roles separately (they are properly versioned and taken care
of, easily available through Ansible Galaxy), but this affects the usage of the
project as a whole (no "stable" releases which makes the use of the project
risky in production, any significant changes need to be carefully coordinated).
Merging everything back together
--------------------------------
Can the different DebOps roles, playbooks, documentation and other software git
repositories be merged into one repository? With a bit of changes, yes. The
current Ansible code within DebOps essentially is one "repository" split into
separate git repositories - you can see it by looking at contents of the
'~/.local/share/debops/' directory after installation. Merging the roles back
with the playbooks shouldn't be a problem, apart from saving the git commit
history.
After the merge, some things could be moved around to improve the directory
structure. Documentation of the roles could be moved to one 'docs/' directory,
and cleaned up to remove redundant documentation like LICENSE files, etc. This
would improve documentation management and allow to use links between different
role documentation without the need to use separate link index files. This
should also resolve the Debian packaging issue.
Having one repository with roles and playbooks could enable easy creation of
stable releases based on branches. The Semantic Versioning could be utilized to
keep a few stable branches, while development is done in the 'master' branch.
The repository could have reserved directories for custom roles or playbooks,
which would allow the users to maintain their own fork and merge any changes in
upstream with relative ease.
Adding new roles to the project would be as easy as currently changing the
existing ones without the overhead of creating new repositories, etc. Any
changes that span multiple DebOps roles could be tracked in one pull request
instead of separate ones for each role, and could be coordinated together.
The ownership of the different roles or code could be managed by the
CODEOWNERS/MAINTAINERS file which specifies which users should review any
changes.
There would be no need to maintain a separate 'debops-contrib' GitHub
organization - third party roles could be prepared and merged in forked git
repositories, or easily maintained in upstream repositories that apply changes
from the DebOps repository.
The use of the Travis-CI for tests is a problem in this model. Running the
whole playbook at once takes too long, and some roles are mutually exclusive
(for example 'apache' and 'nginx'), so using just one test for the whole
project is not feasible. Travis-CI has a "build matrix" feature which allows to
create separate jobs, which could be used to create a set of tests for
different parts of the code, however the limit of maximum 200 jobs, and good
behaviour suggests that this shouldn't be used to test all of the roles
separately, as it is done now. Perhaps the main repository could do just a few
tests for major roles and playbooks (ownCloud, GitLab, Netbox, ie. any
user-facing application) which should test a relatively large part of the
project codebase, and testing of all the roles separately could be done on
a new infrastructure based on GitLab-CI.
Some roles still might be popular enough to warrant their availability
separately through Ansible Galaxy. This could be done automatically by
extracting the selected roles and publishing them in their own separate git
repositories, signed by a bot or a human, depending on number of roles. Ansible
configuration allows to use multiple role directories using a 'roles_path'
configuration variable with $PATH-like syntax, therefore cloning the main
DebOps repository and adding the role directory in the 'roles_path'
configuration variable shouldn't be a big issue. Roles would still be designed
to be self-contained, so this use case will stay valid.
The DebOps scripts will need to be updated to support the new deployment model.
This might be a good enough reason to finally rewrite them from stratch and
update the user interface to support subcommands. In similar fashion, third
party code that was created to support DebOps could be merged in the main
repository as well - example roles, test suite, example playbooks. We can
carefully design the final directory structure to support this.
Final thoughts
--------------
This is just a proposal - I would like to hear the thoughts about this from
other DebOps users before changing anything further. I think that proper way to
do this would be first to review and update the DebOps Policy and Guidelines
(https://github.com/debops/debops-policy/) to reflect the new state of the
project. Then work can be done to merge all existing code into one repository
to ensure that the commit history is preserved. After that, work can be done on
redesigning the existing code, updating documentation, etc.
Cheers,
Maciej
6 years, 7 months
- 5
- 14
[debops-users] Redesign of the Postfix role and DebOps mail stack
by Maciej Delmanowski
Hello everyone.
In case you don't know already, I plan to redesign the 'debops.postfix' role
from the ground up. Current set of changes can be checked out in
a pull-request:
https://github.com/debops/ansible-postfix/pull/93
The goal is to reorganize the DebOps mail stack so that the 'debops.postfix'
role only manages the Postfix configuration files, but not everything else
- that would be done by other Ansible roles which use 'debops.postfix' as an
API. I want to move most of the configuration that current 'debops.postfix'
role supports into a new Ansible role, 'debops.postconf'. It would provide
a way to control what a given Postfix instance does akin to existing "Postfix
capabilities".
The new 'debops.postfix' role is not compatible with the old one, and existing
setups will probably break if not taken care of. You should review your current
Postfix configuration and implement it using the new Ansible inventory
variables. If you find out any issues, let me know in the PR so that they could
be fixed before the final release.
There are other DebOps roles that currently use 'debops.postfix' as
a dependency. These will also be updated along so that they work with the new
infrastructure in place:
- debops.mailman: https://github.com/debops/ansible-mailman/compare/master...drybjed:postfi...
- debops.smstools: https://github.com/debops/ansible-smstools/compare/master...drybjed:postf...
- debops.dovecot: changes are not public yet, but I'll probably refresh the
whole documentation along as well
There will be also new roles added to the stack:
- debops.opendkim: https://github.com/debops/ansible-opendkim
The role is not yet part of DebOps, since it relies on changes in 'debops.postfix'.
- debops.postconf: as mentioned above, it will be a general-purpose
configuration role which will re-implement the existing "Postfix capabilities".
Role not yet written.
- a SPF policy role, not yet written
- a OpenDMARC role, not yet written
Other anti-spam features will be added at a later time.
Due to how current DebOps development process works, I want to have all of the
roles that depend on 'debops.postfix' updated before the main PR is merged.
When new 'debops.postfix' redesign is finally released and tagged, other
Ansible roles will be able to test it properly on Travis-CI and will be
released as well. It would be great for concerned users to test the
'debops.postfix' changes now to find out and fix any issues that might arise.
If you find anything, let me know.
Cheers,
Maciej
6 years, 7 months
- 1
- 1
- ← Newer
- 1
- Older →