Re: [debops-users] Plans for testing DebOps roles

On Jun 04, Nicolas Quiniou-Briand wrote: Interesting topic. I'm actually thinking about testing DebOps upgrades and how to version the project, now that we have stable releases. The things are actually connected, since the version number is supposed to indicate if a given version will work in your environment correctly, or not. I'm wondering how the project could implement proper upgrade testing, from one stable release to the next stable release. It would probably have to involve Vagrant and GitLab CI pipeline, the problem is that the existing one runs for over 7 hours[1], although on a host with slow disk I/O which is the biggest factor. Throwing more hardware at the problem would probably help. I'll see what I can do about it when my development environment has an overhaul. Perhaps this would have to be a separate GitLab CI pipeline (in effect, a separate git repository) that takes the stable branch, newest master, then creates different common setup using stable branch and re-applies the master branch and checks various things like if the GitLab web interface is up, or mail server works, etc. So, another separate test suite that checks for different things than the current per-role tests. As for the version numbering, we're currently using SemVer with major.minor.patch versioning. The changes on the patch level are self-explanatory - bugfixes that don't change the functionality. But what about the major and minor version, when do they change and what is their meaning? I had an idea that minor changes would be the changes that have to be done in the Ansible inventory to make the roles and playbooks work; major changes would be those that require changes on the remote hosts which are not, or cannot, be implemented via roles or playbooks. But this feels too broad for me, I don't want to fall into to browser-itis trap where we now are on version 75 of Google Chrome and 67 of Firefox - are these version numbers relevant anymore? I think that the versioning scheme should reflect something like this: - major version is incremented when large parts of the environment managed by DebOps require a rebuild from scratch. For example, LDAP support was rebuilt recently from the ground up, and existing environments would break without overhauling everything, and these changes were unfeasible to be implemented in the main playbook. It's hard to quantify what "large parts of the environment" are, but I would say if a change requires an big time/work investment from the sysadmin to implement new modifications in the environment, it warrants a major version bump. - minor version is incremented when there are changes in the roles or playbooks that provide new functionality, or require modifications to the Ansible inventory or the project directory to work correctly. Think, roles or variables were renamed, variable types and values changed, etc. The upgrade notes are very helpful in this case and should be detailed enough that the users can deal with the changes without disruption of their production environments. - patch version is incremented when new bugfixes are committed from the master brnch. The releases shouldn't be too frequent, say every week or so if there are pending changes in the stable branch - that way multiple bugfixes in a short amount of time have the chance to be tagged together and the patch number will not go up rapidly during the lifetime of the stable release. With the current schedule for new stable releases every 4-6 months, the major version will probably be incremented 5-10 times until the project stabilizes enough, after that I would estimate major release every year or two - perhaps we could then keep the major version in lock-step with Debian releases. Any thoughts on this scheme, before I commit it in the documentation? [1]: https://gitlab.com/debops/debops/pipelines/64285334 Currently tests use partially static and partially dynamic environment; DNS and hostnames are managed dynamically. I'm afraid of setting up completely static test environment, because then DebOps would have to work against that specific environment and not against anything users throw at it. Throwing some more randomization into the tests wouldn't hurt, I imagine. Hmm, ruby... :) At least Testinfra is in Python, so we could stick to one language instead of two. Cheers, Maciej