On Jun 04, Nicolas Quiniou-Briand wrote:
On 2019-05-31 3:08 p.m., Maciej Delmanowski wrote:
>How is the experience so far? Do you use the stable release, or stick to the
Fine. I use both. I'm testing DebOps with Vagrantfile from GitHub repo and I
use stable branch to run on production servers.
Interesting topic. I'm actually thinking about testing DebOps upgrades and how
to version the project, now that we have stable releases. The things are
actually connected, since the version number is supposed to indicate if
a given version will work in your environment correctly, or not.
I'm wondering how the project could implement proper upgrade testing, from one
stable release to the next stable release. It would probably have to involve
Vagrant and GitLab CI pipeline, the problem is that the existing one runs for
over 7 hours, although on a host with slow disk I/O which is the biggest
factor. Throwing more hardware at the problem would probably help. I'll see
what I can do about it when my development environment has an overhaul.
Perhaps this would have to be a separate GitLab CI pipeline (in effect,
a separate git repository) that takes the stable branch, newest master, then
creates different common setup using stable branch and re-applies the master
branch and checks various things like if the GitLab web interface is up, or
mail server works, etc. So, another separate test suite that checks for
different things than the current per-role tests.
As for the version numbering, we're currently using SemVer with
major.minor.patch versioning. The changes on the patch level are
self-explanatory - bugfixes that don't change the functionality. But what about
the major and minor version, when do they change and what is their meaning?
I had an idea that minor changes would be the changes that have to be done in
the Ansible inventory to make the roles and playbooks work; major changes
would be those that require changes on the remote hosts which are not, or
cannot, be implemented via roles or playbooks. But this feels too broad for
me, I don't want to fall into to browser-itis trap where we now are on version
75 of Google Chrome and 67 of Firefox - are these version numbers relevant
I think that the versioning scheme should reflect something like this:
- major version is incremented when large parts of the environment managed by
DebOps require a rebuild from scratch. For example, LDAP support was rebuilt
recently from the ground up, and existing environments would break without
overhauling everything, and these changes were unfeasible to be implemented
in the main playbook. It's hard to quantify what "large parts of the
environment" are, but I would say if a change requires an big time/work
investment from the sysadmin to implement new modifications in the
environment, it warrants a major version bump.
- minor version is incremented when there are changes in the roles or
playbooks that provide new functionality, or require modifications to the
Ansible inventory or the project directory to work correctly. Think, roles
or variables were renamed, variable types and values changed, etc. The
upgrade notes are very helpful in this case and should be detailed enough
that the users can deal with the changes without disruption of their
- patch version is incremented when new bugfixes are committed from the master
brnch. The releases shouldn't be too frequent, say every week or so if there
are pending changes in the stable branch - that way multiple bugfixes in
a short amount of time have the chance to be tagged together and the patch
number will not go up rapidly during the lifetime of the stable release.
With the current schedule for new stable releases every 4-6 months, the major
version will probably be incremented 5-10 times until the project stabilizes
enough, after that I would estimate major release every year or two - perhaps
we could then keep the major version in lock-step with Debian releases.
Any thoughts on this scheme, before I commit it in the documentation?
>Yes, I'd like to improve the role testing in the Gitlab CI
pipeline by adding
>a testinfra checks to each tested role. I'm wondering if these tests could be
>generalized so that thestinfra can be applied to an actual set of hosts
>deployed somewhere, in which case I would probably make a new 'testinfra/'
>directory in the root of the DebOps monorepo and work on tests there.
According to Testinfra's docs , it looks like it's possible to test
multiple hosts. Moreover, there is an Ansible backend  able to get
informations from inventory.
Currently tests use partially static and partially dynamic environment; DNS
and hostnames are managed dynamically. I'm afraid of setting up completely
static test environment, because then DebOps would have to work against that
specific environment and not against anything users throw at it. Throwing some
more randomization into the tests wouldn't hurt, I imagine.
>Buf if that's not feasible or useful, I suppose that a
'tests/' subdirectory in each
>role directory could be used instead, but that could be complex if later on in
>the pipeline multiple roles would have to be tested together. Something to
>think about, I suppose.
Did you consider other tools for testing Ansible roles ? From my point of
view, InSpec is an interesting tool with an important set of resources .
I saw that Robin is part of dev-sec.io organization on GitHub, I'm sure he
has an opinon on it :-)
Hmm, ruby... :) At least Testinfra is in Python, so we could stick to one
language instead of two.