[debops-users] News from the DebOps project

Scott scosist at asphaltzipper.com
Tue May 1 22:43:20 CEST 2018

On 04/16/2018 03:37 PM, Maciej Delmanowski wrote:
> Hello everyone,
> Welcome from the rainy GdaƄsk, Poland, to another irregular status update
> about the DebOps project. This time I want to talk about where the project
> stands after the migration to the monorepo and what I want to work on next.
> Untangling the mess
> -------------------
> Since the latest release, I have been working on untangling the mess that are
> the 'debops.auth' and 'debops.console' roles. They were one of the first roles
> in DebOps, thought out to be aggregates of sorts for different areas of host
> management - authentication and authorization for the first one, and "Linux
> console stuff" for the second. Over time I realized that doing aggregate roles
> like these doesn't make much sense, when you can achieve the same by utilizing
> Ansible playbooks - technically both types of aggregation result in the same
> set of tasks, and having more granular roles lets you mix and match different
> contexts as you see fit.
> Therefore for the last few days I have been working on splitting the above
> roles apart. You can see the results in the new set of roles in the 'master'
> branch.
> The 'debops.netbase' role, extracted from the 'debops.console' role lets you
> configure '/etc/hosts' and '/etc/networks'. In the future, it could also take
> over '/etc/services' to handle all the configuration managed by the 'netbase'
> Debian package, however I didn't want to distrupt everything just yet. Since
> '/etc/services' file needs to be mangled somewhat to properly mesh the DebOps
> changes with Debian base, that's still something to think about. Perhaps
> utilizing 'libnss-db' as a database of DebOps-managed services could be
> a solution, although I'm not sure. An alternative in a larger cluster would be
> providing additional service information via LDAP.
> The 'debops.sudo' role, extracted from the 'debops.auth' role lets you
> configure 'sudo' service. Since the '/etc/sudoers.d/' directory exists,
> I don't think that other roles will use 'debops.sudo' role as a dependency
> that much - creating a configuration file directly is much easier. However
> having one place to configure global 'sudo' defaults seems like a decent idea.
> The 'debops.system_groups' role, extracted from the 'debops.auth' role,
> finally somewhat formalizes various UNIX system groups used by DebOps, like
> 'admins', 'sshusers' and 'sftponly'. They are properly documented, and role
> allows for easy addition of new ones, as well as creation of simple ACL for
> UNIX groups that can be used by other DebOps roles.
> Having the above two roles as separate from 'debops.auth' allowed me to
> reorganize the contents of the 'debops.bootstrap' role and replace parts of it
> with the new roles. This allows for better idempotency within the project and
> keeps configuration of different services in their respective place.
> The things left in 'debops.auth' are a few services focused on password
> management, like libpam-cracklib, which will need to be replaced with
> something better suited for LDAP environment. There's also a few different
> things related to LDAP authentication and authorization. These will need to be
> split into smaller roles in the future. I would like to refresh the
> 'debops.slapd' role a bit and add replication setup before I'll mess with LDAP
> stuff though.
> The 'debops.console' role contains a few tasks related to filesystem mounts,
> which should be moved to a separate role. I'm not sure where
> fstab-related role could be placed in the 'site.yml' - having it before
> network configuration will make NFS and other network filesystems broken
> during initial configuration, having the filesystem mounts too late in the
> playbook might hinder configuration of shared filesystems for different
> applications. Perhaps two or three roles would be needed for this.
> The other thing in 'debops.console' left to handle is the "fsckfix"
> configuration, which requires modification of kernel boot parameters via
> 'debops.grub' role to properly configure fsckfix under systemd. However
> 'debops.grub' role is currently a bit ill-designed, and needs to be cleaned
> up[1]. I'll get to it in due time.
> [1]: https://github.com/debops/ansible-grub/issues/8
> Python 3 compatibility
> ----------------------
> The DebOps scripts should be now mostly compatible with Python 3. There are
> now source and wheel Python packages available on PyPI built to work with both
> Python 2.7 and Python 3.5+, so the project should be covered for full
> transition to Python 3 installations only. There are slight issues here and
> there, but thanks to the issue reports and PRs from the community they are
> slowly weeded out. Thanks everyone for helping along. :-)
> This unfortunately is not yet the case with the environments managed by
> DebOps. Almost all DebOps roles use 'python-*' APT packages when needed, which
> result in Py2 compatibility only; Ansible local facts that are written in
> Python use '#!/usr/bin/env python' which results in the facts not working
> correctly when Python 2 environment is not available.
> To remedy that in the long term, I finally created the 'debops.python' role
> which manages multiple Python versions at once. In the future I plan to
> install any Python APT packages via this role as a dependency, which will
> allow to selectively disable Python 2/Python 3 support as needed. I'm not sure
> if Debian hosts without Python 2.7 installed are entirely possible, but soon
> we will be able to find out.
> The Ansible local facts and some other scripts used by roles internally have
> a slightly different issue. Due to the use of 'pycodestyle' for PEP8 linting
> of the scripts, shebang lines cannot be templated by Jinja. The solution I'm
> considering is to provide two versions of the same script, one for the Python
> 3 environment, and another for Python 2 environment. You can see an example in
> the 'debops.python' role. I suspect that most of the scripts would be the same
> between two versions, therefore keeping both instances updated should be
> relatively easy to do.
> Unprivileged LXC containers
> ---------------------------
> The 'debops.lxc' role hasn't been refreshed for a while, apart from the
> changes in the shell scripts. Since Debian Wheezy will be EOL soon[2], it's
> time to drop the cruft that was needed to setup LXC on Wheezy via
> 'jessie-backports'.
> One other thing that I would like to implement is unprivileged LXC containers.
> I've found a guide which implements them on Debian Stretch[3] which seems
> pretty easy to implement in DebOps. One obstacle is to find a way to sanely
> handle subuids and subgids. The 'root' account would need to have one set of
> subuids/subgids so that unprivileged LXC containers created by root and
> started at boot time would be possible. I wonder if one set of subuids/subgids
> shared among different LXC containers which would use essentially the same
> users/groups on the host would be a good way to handle this, security wise.
> Alternatively, the root account would have to have multiple sets of
> subuids/subgids for each unprivileged LXC container which quickly becomes an
> implementation problem. Perhaps there is some project already that can help us
> handle this?
> [2]: https://wiki.debian.org/LTS
> [3]: https://github.com/BenSartor/unprivileged-lxc-containers
> Reorganization of DebOps Policy
> -------------------------------
> The old project documentation still needs to be moved into the new
> documentation structure. One particular section is the DebOps Policy and
> Guidelines[4] which formally defines how DebOps project works, how Ansible
> roles are written, etc. I think that restructuring it into one unified section
> of the documentation with numbered and easily referenced sub-sections might be
> useful. This could then be used by 'ansible-lint' tests to reference relevant
> parts of the DebOps Policy.
> [4]: https://docs.debops.org/en/master/debops-policy/index.html
> Public "work in progress' branches
> ----------------------------------
> I recently published some "work in progress" branches in my fork of the DebOps
> project on GitHub[4], named as 'wip-*'. If you like the idea, let me know
> - I can publish more stuff, after cleaning up the private data used for
> testing. The published code won't necessarily reflect the final roles and
> playbooks, but I'm trying to have working solutions where possible.
> [4]: https://github.com/drybjed/debops/branches
> Ongoing development
> -------------------
> There are of course things that need work which I mentioned previously
> - Travis-CI test suite needs to be refreshed, GitLab CI tests need to be
> expanded to actually verify that the roles configure the applications
> correctly, there are some monitoring roles like Icinga setup that is really
> overdue, etc. This will be done as I go. DebOps has multiple areas of focus,
> some of them are properly fleshed out already, others not so much. If you feel
> that you have a good idea which could be implemented in the project, either
> for the DebOps environments via Ansible roles, or in the monorepo itself as
> the stuff that supports the project itself, let me know.
> Thanks for your time and remember to have fun.
> Maciej
> _______________________________________________
> debops-users mailing list
> debops-users at lists.debops.org
> https://lists.debops.org/mailman/listinfo/debops-users
Great to read about the upcoming changes. While reading through the
documentation today to I ran into a couple of errors and realized it was
due to using the latest documentation with an older version of DebOps.
I've yet to upgrade to the latest version until I get a chance to fully
review the changes. I'm hesitant to upgrade until I know it will be
smooth as this is in production, especially with the X.509 certificate
changes (although it appears that only affects new certificates).

To my point, I wonder if it wouldn't be worthwhile to have separate git
branches for each release; or is it too early for this (perhaps this is
more suitable for major point versions)? I imagine it would be helpful
whilst running an older version and preparing to upgrade as well as
tracking changes and being able to more easily view version diffs.

Thanks for all your work. Looking forward to the new improvements.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.debops.org/pipermail/debops-users/attachments/20180501/be9fe284/attachment.html>

More information about the debops-users mailing list