Re: [debops-users] RFC: Merge all DebOps git repositories into one

Hi, I've removed all the parts I generally agree with and have no further comments. Within this mail I've proposed my alternative solution to most of defined problems (as I understand them). The upstream version doesn't need to. But it's kind of an opportunity to stabilize DebOps. As I understand it: - You release some tagged upstream version (say 1.4.0) when most of the things works. In general it should work with the current... Debian Testing. - it gets included/updated in the testing Debian - Testing Debian goes to "freeze" state - and from this moment onwards you won't be able to change the debops within Debian except for fixing bugs - so you have a stable branch 1.4, and can publish new 1.4.1, but 1.5 won't get into Debian stable at all. - Upstream can be changed, released etc - it's not bound by the Debian cycle, but won't get into Debian until new Debian Stable release - hence you get a stable release - which has benefits. If my stable debian-reviewed debops still manages correctly my debian-reviewed stable nginx version - I'm fine with it. Only when I need some function that is not there I'll be chasing newer versions. Also - newer upstream version may, or may not work with frozen Debian nginx version (well - it should. But I'm not so certain about webapps like owncloud). Yeah, that exactly that. At least if DebOps is going to "stabilize". Some of the described burden is lifted by my idea described below without a merge. Maintainer would be able to stick to the single repo. Multiple "stable" versions could be maintained and separate flavors used. (snip my own answer as it better described lower) And I believe it's correct as far as the playbooks don't "age" too much during Debian release cycle. So - limit dependencies on external packages/versions if there are any for core playbooks. I'm beginning to understand and feel you problem a bit better. Let me offer you a more detailed example of how this can work with split repos and Debian - and a path for stabilizing DebOps (which might not be what you want anyway, but stick with me for a minute). Let's forget about tarballs for a second. From the Debian project perspective "upstream" is whatever you decide to publish. Most of the time they are still working with the tarballs but what they really want is: - a single source of truth - a version (relevant afaik: https://wiki.debian.org/PackagingWithGit) If you manage to: - Provide a single main versioning for the DebOps project (separate from role/tools versioning). Connected to the single git repo. - Store a verified info in the main repo about all the required parts - e.g. a yaml file with list of repositories, and the last verified SHA-1 checksums. - Sign this yaml file with hashes - instead of signing separate repos. - During debops-update VERIFY the file before cloning repos. Clone to the specified SHA1 only. (This is a security improvement over current approach I believe) - Allow people to use some branch instead of SHA1 on request. And add additional repos to the list so that debops automatically pulls them. - And finally: Provide a build script inside the main repo which: - pulls verified repos up to the verified state, - builds the docs, - creates a tarball with version in it's name which is reproductible (https://reproducible-builds.org/ (a given state of main repo will always produce THE SAME tarball - even if some branches if other repos change). So I got back to the tarballs. Tarballs can have a version control (kind of) but it requires you to periodically build the project (from validated by you sources) and then sign the resulting tarball. The connection from the repo and it's SHA1 to the tarball is direct and reproductible. I believe this: - Does solve Debian problem because they can either use your last produced tarball (github independent) or use your single GIT repo which has a "compilation" process which calls github in the meantime. - Shifts your signing issue from signing repos to maintaining a signed list of commits. So no easy solution. Procedure: * Maintainers handle their repos themselves. You don't even need access. * When they believe they are "ok" - they create a PR incrementing the hash in main repo. (so you observe the PR only on one repo) * You review the PR by checking diff between versions and dropping or accepting AND SIGNING the change. - Validates the code better. In my current default debops setup I have 23 NOT SIGNED repositories at all - and I run their code. for i in $(ls); do pushd $i > /dev/null; git tag --verify $(git describe --abbrev=0 --tags 2>&1) 2>&1 | grep 'Good signature' > /dev/null || echo $i error ; popd >/dev/null; done - Allows you to fix previous stable versions: - checkout the role at commit from which Debian version was created - fix the bug and tag the fixing commit. - checkout the main repo at release-commit and: * increment the fixed role hash, * sign, increment main version bugfix - Reproductibly-build the tarball with with only required changes and update Debian package. - In semantic versioning: * You keep the major number if major numbers of all roles didn't change (no incompatible API changes) - otherwise increment major, zero minor. * You increment the bugfix version if the update fixes bugs without including/changing features. * You increment the minor version otherwise (function change, not bugfix, but API stable). Instead of the file signing you can probalby sign repo and use submodules - but that's less explicit, more opaque and doesn't survive creating a tarball. And there's even now a hole if I understood the process correctly. debops-update doesn't validate your signatures at all - so attack on gh repo might alter code running as an administrator if I'm not mistaken. With a described central main "debops" with list of SHA1 this is possible too. There can be testing/stable branch - pulling different versions of roles (and different sets of roles) - and the same there can be more than one single repo. You would be able to create your own mix of roles, validate as you see fit and sign. Debops update would accept the file as long as the key would be trusted. Two-way synchronization can go awry, but would be possible. Ok, so both approaches would require writting some additional code. ;) That's a bit what I was going at too. This human-project-lead - would have to consider only one repo and PR to only one repo (except for occasional git diff to verify if role maintainer is not including a backdoor). True. But can be updated to clone from gitlab by repo name and with possibility of "freezing" debops version as a verified tarball the dependency on GH is less of a problem. Cheers, -- Tomasz bla Fortuna jid: bla(at)thera.be pgp: C009FC9BD59B22FE @ pgp.mit.edu