9:53 p.m.
Good evening all
Sorry for the delay.
At first I was not convinced about this merge. Now that I went thought
the discussion and thought about it I agree that this will allow us to move forward.
ganto already said it: Much work is put into roles. They are the key
components. But the real value of DebOps is that they all work together and I
think in that regard such a merge would help to reduce the maintenance
overhead, primarily for drybjed.
On 08/15/2017 08:36 PM, Maciej Delmanowski wrote:
On Aug 15, Reto Gantenbein wrote:
As for the role being "interesting" enough to be published separately on
Ansible Galaxy - this process will become highly automated, since the roles
would not need to be tagged with GPG signatures (the original code will be
signed and present in the main DebOps repository). If demand is high enough,
all of the DebOps roles could be published that way on Galaxy using a bot.
What will change is the direction the code is sourced from - instead of
pulling all of the separate roles together into one directory structure, roles
will be extracted from one "source of truth" git repository into separate
satellite repositories.
That is pretty much what I wanted to reply to an earlier mail. Just script as
much of it as possible and then it won’t matter if a few or all roles are up
there.
> There are many other advantages as already nicely summarized by drybjed
> in the initial mail, I don't have much to add here. Maybe one point
> that I would like to mention that for new users/contributors it might
> be much easier to grasp the idea of Ansible roles and DebOps if they
> can easily read and run an individual role instead of a huge repository
> with hundreds of roles and playbooks. At least I got this impression
> when trying to dig into openshift-ansible [1] which is meant to setup a
> OpenShift container PAAS and also consists of dozens of Ansible roles
> with lots of possibilities and interpendencies.
I don't think that the development model (one repository vs roles in separate
repositories) changes this. At the end of the day, DebOps is yet another huge
repository with hundreds of roles and playbooks, just laid out differently.
New users still feel overwhelmed with the project, and need to learn it before
trusting it. This is a documentation issue, and with a series of tutorials
with encreasing levels of complexity should be relatively easy to overcome.
Agreed. Users could still start with one role, review it, run in
standalone and so one.
> Stability:
>
> DebOps definitely should have a way to "freeze" and properly test and
> package a state of the Ansible roles which can consumed easily by
> sysadmins who prefer to focus on application and architectural issues
> and not bootstrap and configuration management issues. The current
> distribution model of DebOps with updating the roles through git and
> the constant rate of changes and redesigns of roles doesn't help to
> build confidence. DebOps should help to make things easier for its
> users not more complicated.
In the curren state this "freeze" would need to be handled in the
'debops-playbooks' repository. Since the roles in separate git repositories,
only mechanism that is valid to use to freeze on specific role versions is
Ansible Galaxy 'requirements.yml' file, which would specify all of the roles
and the desired versions.
Let's say that DebOps offers two "stable" branches, 1.x and 2.x, each one
with
a frozen set of roles. there's also separate 'master' branch which gets
updated before being frozen. Now, a 'debops.nginx' role has some updates and
a new version is released, now that version needs to be updated in the
'debops-playbooks' 'master' branch. And then 'debops.sshd', and
'debops.gitlab', etc., etc. Each release of a role requires a parallel update
in 'debops-playbooks'. Note that right now an update in the playbook
repository is needed only when a new role is added, or a playbook of an
existing role is updated; otherwise 'debops-playbooks' is not changed all that
much.
Without changing the development model of the project and implementing
"freezes" like the above, that would put even more pressure on the project
maintainers and add even more work. Can this be automated? Not without
automating commit signing...
The reason for this is that we are essentially duplicating the internal 'git'
repository mechanisms using Ansible Galaxy 'requirements.yml' file. How would
that change if everything is in a single repository? Any updates to a role are
in the 'master' branch of the main git repository, there's no need to track
role versions - in fact I'm not sure if role versioning would be needed at all
in this case, at least not as frequent. The main git repository would be
versioned as a whole, and exported DebOps roles on Ansible Galaxy could use
the same version numbers. Export could be easily done by a bot, the roles
published on Ansible Galaxy IMO don't need to be signed because a) they are
already signed in the main DebOps repository, and b) 'ansible-galaxy' command
downloads tarballs, not git repositories.
Also exports should be signed in the end.
a) Users would need to run a manual diff against the Galaxy "export" and
the main tree so that they can trust the unsigned export.
b) Currently not done but there is an proposal about this
https://github.com/ansible/proposals/issues/36
Signed role "exports" would only happen on releases which are signed
anyway. Everything else is just automation.
Not singing exports is the chicken and egg problem again. Just sign it
even if the tooling or the need is not yet apparent to everyone. Related (goes
in the same direction): https://github.com/debops/debops-tools/issues/170
With one git repository there wouldn't be dozens of Changelogs to track, but
only 1 Changelog. Stable releases would be required to not do any changes that
break existing setups. Perhaps, with time, DebOps would also offer some way to
migrate between different releases. Also, with much smaller number of git
repositories to handle, detailed release notes are feasible to write.
Users might not use all parts/roles of DebOps just yet so the changelog
should be easily filterable.
> In a distributed world this CI machine would then also be responsible
> for branching and tagging the individual repositories appropriately as
> it is required to create releases and bugfix channels.
Tracking all these repositories correctly quickly becomes a nightmare, and you
are essentially reimplementing git to do this. Wouldn't it be better to use
the existing git mechanisms (branches, tags) properly?
> Consistency:
>
> There are already a nice bunch of policies and guides available for
> DebOps but they have to be read and understood and there is no
> automation available which checks and enforces those. With ansible-lint
> [2] and ansible-review [3] there are already some tools available which
> could be leveraged to improve and maintain consistency in this large
> number of roles.
That's a good idea to implement in the tests. +1.
I guess we all agree on this and it is not that such tools are not
already known/available nor used by us. One of them is [yaml4rst] which enforces
defaults/main.yml format. The script which ties it all together is
[debops-optimize] which even uses ansible-lint. Sure, this will need to
become an official part of DebOps to be more visible.
[yaml4rst]: https://pypi.python.org/pypi/yaml4rst
[debops-optimize]:
https://github.com/ypid/ypid-ansible-common/blob/master/bin/debops-optimize
> These were some of my thoughts. If some effort regarding packaging
> should be made, I'm definitely willing to help as I'd love to see a
> DebOps package in Debian (or at least on my servers).
Thanks for your thoughts! I would also like to see DebOps as an official
Debian package in a stable release. Unfortunately I don't think that is
possible with the current way the project is maintained, so I'm willing to
shake that up to see it happen.
Agreed, this would be awesome to see and we will get there eventually
:)
> - The role maintainer model has to be given up (at least
privilege
> wise). Either one is a main developer of DebOps and has access to
> everything or one has no rights.
On the contrary, with just one repository, you can fork it on GitHub and gain
access to everything that way. Your changes could then be pulled to the main
repository from your fork. If your fork is popular enough that users start
using it instead of the main DebOps repository, you win.
I agree with ganto. Write privileges could only be given on a all or
nothing base. Some git receive hook should be put in place to limit this.
I was fine with having no write access to certain parts of the
project because I don’t/did not need them (principle of least privilege).
> - It's harder to follow development and see issues of
individual roles
> as everything will be mixed together.
I think that this ties with the case where multiple roles need to be modified
together to work properly (think project-wide firewall changes, or the recent
Postfix updates). In the current model, each role requires a separate PR to
handle the required changes, in one repository model all the needed changes
could be grouped together in one PR, developed over time to encompass
everything that's involved in that PR, and committed at once.
Looking at issues for individual roles could be handled by properly using
labels and/or tagging issue titles. In the current model I stopped using
GitHub issue labels altogether because each repository needed to have them set
up separately. I guess that this could be handled by writing scripts that
manage labels in all DebOps repositories via GitHub API, but that never
happened (lack of manpower, I suppose).
There are both pros and cons here. I just want to emphasize one con for me:
It would be much more difficult to watch only parts of DebOps. For example, I
maintain debops.cryptsetup and a few others and would need to watch for issues
and PRs.
Getting notified for all PRs/issues of DebOps directly does not scale.
I guess the easiest solution would be a PR/issue template where contributors
need to CC the role maintainer. That should solve it.
manage labels
Scripts for this exist already. I think I starred a few of them when I was
looking for something. Not sure what I ended up using.
To conclude:
I think the advantages overweight the disadvantages. I am fine with DebOps
moving in that direction.
When the merge is done, I would like to do a review in that sense that nothing
of the role code was changed from the individual git repos (which I reviewed and
implicitly signed) to one repo and then create a signed commit. This way, the
merge which might be difficult for others to review is at least signed by two
independent parties. Scripts should be used for the merge where possible and it
would be nice if those scripts would also help with reviewing an already completed
merge.
I just saw that drybjed already initialized the new repo
(https://github.com/debops/debops). Now the rename of the debops tools to
debops/debops-tools pays off :)
--
Live long and prosper
Robin `ypid` Schneider -- https://me.ypid.de/