Hi Maciej, thanks for the great reply. I hope that others on this list
will chip in as well.
Sorry for the late response by the way, I am way too much of a
perfectionist to be able to slam an email out of my brain just like
that. But I'll surely miss something no matter how hard I try, so here
goes...
On Mon, 2020-07-06 at 23:23 +0200, Maciej Delmanowski wrote:
> Also, some battle history, cuz' Maciej is a sucker for war
stories.
Indeed, I am. :-)
Pay attention y'all! The man says he wants *war stories*, so give him
all you got!
The "ginas" name was a backronym, from "ginas is not a
server" -
since it was meant to manage a cluster from the start, contrary to
other such projects being worked on at the time (2013) which focused
on everything done on single hosts. But, a better name came along and
I switched it.
DebOps is a better name indeed. But let us not forget the ginas
genesis, for it be our lexical legacy.
...or something like that ¯\_(ツ)_/¯
I think that Docker is a great solution in a narrow field of
stateless applications, as long as you have everything that is needed
underneath Docker for it to work under control. There are of course
solutions for that like Docker Machine (more for development),
Rancher, now Kubernetes and other orchestration platforms, but these
are great when you use them as managed services. When you try to host
them yourself, the problems of setting everything from the ground up
come back again. And without solid foundation, it's only a matter of
time when something breaks under Docker and everything comes down.
Building everything properly takes time and effort.
Yeah, I made a few painful design decisions with the Docker
infrastructure at Bits of Freedom at the time. I still feel bad about
my successor having to clean it all up.
> Long story short, I turned off the very last legacy,
hand-managed
> VM last week.
Congratulations! :-)
Thanks a lot! It was an awesome and fulfilling journey. Which will not
be over for a long time, thankfully. There's always more to be
automated.
> [..]
> And the best thing? It is *all* automated.
This is very impressive! I suspect that you had to write at least
dozen additional roles to manage things that are not yet in DebOps
itself, but some of that effort could probably be offloaded to
already existing roles like nginx, ferm, sysctl, postfix, etc. That's
what I was aiming for with the project from the start and I'm very
glad that you were able to make it work.
Something like a dozen roles, yeah. You'd be surprised about all the
things I managed to do with some clever variable tricks and existing
DebOps roles. Some of the roles I wrote for CipherMail (built to be
compatible with DebOps) can be found here:
https://gitlab.com/ciphermail?filter=debops
I would love to see some of these things in DebOps someday - putting
roles upstream will definitely benefit other users and will make the
code resilient and easier to maintain when big changes happen. Of
course dumping everything at once will probably break some things,
and you probably would want to clean up secrets or stuff meant only
for your own environment, so it will take time.
Still, if you plan to contribute some of your solutions to DebOps,
that will be awesome.
You bet that integrating these roles into DebOps mainline are very high
on my whishlist. You'll be seeing PRs for these roles in the coming
months, starting with the largely rewritten debops.dhcpd role.
Did you just found a way to beat
https://xkcd.com/1319/? I guess
sharing automation solution between organizations will do it...
Haha, maybe so! And OMG that title text is the best!!!
It's great to hear that a project that I started long ago helped
you
in such a way. This tells me that my decision to make it open by
default and work on it essentially "from outside in" to ensure that
other people would be able to mold it for their own needs was a
correct one.
Which is exactly the reason why I decided to give that project a try.
Thanks a lot :)
Of course don't forget that DebOps is just essentially an icing
and a
cherry on a very big cake which is other Open Source and Free
Software projects, especially the whole Debian ecosystem, whthout
which DebOps probably would not exist today, and Ansible which is THE
tool which made such a thing possible in the first place. I'm not
sure if DebOps could be recreated using other configuration engines
such as Puppet or Saltstack - maybe, maybe not.
You're absolutely right and I am well aware of the importance of free
and open source software. I'm glad to be working for a company that
makes (mostly) FOSS, which in itself is also based on other open source
projects.
I was aware pretty early that making DebOps an open project worked
on
by lots of people from different organizations will eventually
require some kind of stewardship over it to keep it going. Usually
projects like these end up with a foundation of some sort behind them
(Apache Foundation, Mozilla Foundation, GNOME Foundation, I can keep
going...), or a separate corporation is formed to perform the role of
a steward (Software in the Public Interest, Software Freedom
Conservancy). I always thought that at some point DebOps could join
the Debian project in some way, not just as a package in the
repositories, but as a platform to give users as a stepping stone for
"what to do after installing Debian". But I feel that it is still too
early for that to happen, there are lots of things to clean up and
improve. Some roles haven't been revisited in years due to lack of
time or user interest.
DebOps joining Debian would be a great move. I'm really not sure when
it would be the right time for that, so we'll better start preparing I
guess ;)
I discussed DebOps with a Debian developer (a project member with many
rights regarding the Debian project) during the launch party for Buster
in Amsterdam. He was interested but was not too familiar with
automation tools like Ansible, so this unfortunately didn't result in
fruitful cooperation.
I'm contacting a few people involved with the Debian project about
this. Let's see what happens.
At the moment DebOps is in a quasi-like state where more and more
people start depending on it, but the project itself is not really a
legal entity. I'm currently covering the bills for the DNS domain and
VPS hosting where various sevices are kept, like this mailing list
for example. These costs are manageable right now, so I'm not worried
about requiring any donations, especially that this would require me
to set up some kind of company to handle taxes in Poland, and other
such things I'm currently not in a capacity to handle myself.
If setting up something like a foundation works the same way in Poland
as in the Netherlands (having founding documents notarized for a
substantial price, needing to find a secretary and treasurer, as well
as handling taxes), then I can strongly imagine that this is not
something that you'd desire. I'll leave this decision to you.
Sponsoring should be fine though. Hmm...
Various other services like Travis-CI, GitHub, GitLab CI,
ReadTheDocs.org provide a free tier which seems to be sufficient for
now. DebOps is intentionally designed to be as much self-contained as
possible - users are able to set up their own GitLab CI pipeline to
faciliate the tests, code and documentation linting performed on
Travis-CI for each pull request can be done locally as well after all
needed tools are installed (need to make this process easier). I
intentionally don't want to depend on any specific SaaS or cloud
service to keep the project self-contained and focused on
selfhosting.
Now I feel bad about not using the debops.gitlab role to manage our
internal GitLab installation. I really contemplated it, but the Omnibus
package is just so damn convenient :/
First of all, thanks for starting this thread! I know that I'm
really
bad at mailing list discussions, I still need to work on some kind of
initiative skills... After the mailing list server was upgraded to
Mailman 3 I hoped that the rate of discussions here would increase a
bit, so it's good to see such an e-mail.
It's always nice to see your work pay off :)
(not sure if this is due to the new mailing list software, but anyhow)
I will have to write down some messages around current problems I
see
in various roles in the project, but usually I prefer to just work on
fixing them instead. I'm afraid that this impacts the development
culture around DebOps - not much discussion about problems and how to
fix them collectively. Perhaps this can be improved in the future.
Thanks for raising this issue. Contributor engagement is something that
a lot of open source projects seem to struggle with. Which is
definitely not saying that there isn't enough engagement amongst DebOps
contributors, but like you are indicating, it's often about engaging in
very specific tasks that are necessary to bring the project as a whole
further. Writing our challenges down for all to see may be a good first
step.
This reminds of rule 3 of the Debian Social Contract: "We will not hide
problems".
> Is there anything you would like to see improved?
I would like to hear the answers to that question as well! One
problem I can definitely see impacting the project is keeping the
codebase secure and trustworthy. I tried to solve this problem by
limiting the number of people that can merge pull requests to the
main repository, requiring (but not enforcing as of yet) signed
commits to allow code authorship tracking, carefully reviewing each
pull request and trying it out when needed before merging, but I
think that this made the development process very slow, some pull
requests are waiting years to be resolved. Sorry about that.
It's a really good thing if there are at least two contributors with
write access to the mainline repository. Those are you and Robin right
now, so that should be alright. Robin has, as far as I can remember,
not approved any of my PRs though, those were all yours.
I'm not really sure how to solve this problem yet. Most likely
more
people would have to be involved in the review and testing process,
with some kind of trusted relationship formed between those who can
merge PRs into master.
I guess a real life meeting of some sort would be a first step
towards that, to estabilish such relationships Web-of-Trust style,
similar to how Debian does it. They seem to handle much larger
project just fine this way, and probably are a good example to
follow.
I'm all in favour of meeting each other in real-life. Let's make that a
major milestone for us in 2021, possibly sooner. I have doubts if the
Chaos Communication Congress is happening this year, but if it is, then
that would be an awesome opportunity for this.
Now that I think about it, DebOps actually is a combination of two
things - the existing Ansible and Python codebase which lets you
manage infrastructure, and a virtual "blueprint" of an IDEAL
infrastructure the project strives to implement - what software to
use for various services, the configuration order of different
services, what protocols to use, etc.
Perhaps such a "blueprint" could help with development and design of
DebOps codebase and provide guidelines for role authors to follow. I
know that there are some beginnings of it in various parts of the
DebOps documentation that describe how the roles should be written,
etc. but that's probably not enough to be such a blueprint. I've
meant to write an Admin Guide in the docs to describe to how to set
up an infrastructure with DebOps but it still hasn't happened yet.
Imre, you seem to have done a very good job without it - perhaps a
more in-depth guide on how you modeled your infrastructure and what
choices you made along the way would be a good starting point?
It's funny, I've discussed writing about our virtualization and email
infrastructure automation with my employer a couple of days ago. It
would be something for our technical blog, but it should be easy to
modify for usage as documentation for DebOps.
Our clustered, highly available email infrastructure is extreme
overkill for the amount of mail that flows through it. That's because
we're trying to eat more of our proverbial dog food. It would however
be a nice showcase of what's possible with DebOps.
I'm in a position where I have been actively using DebOps at my
workplace for several years and essentially doing my work "through
it" as much as possible.
I definitely don't plan to stop that any time soon - I don't see
myself going back to managing Linux infrastructure entirely by hand
again, and switching to a different project like this is probably
impossible - I don't see any modern alternatives that would handle
the scope that I need. So, in essence - I'm stuck with you for the
forseeable future. ;-)
That's great to hear :)
The same applies to me, every change I make revolves around DebOps
roles and playbooks. I'm absolutely hooked.
But if I stop for any reason, I plan to give reins of the project to
somebody else, of course. Who will that be, I'm not sure yet. I'll
probably look closely at the list of contributors to the project[1],
not only at the number of commits but at recent activity as well. Or
perhaps over time we can form a larger group of people that can
maintain the project as a team.
Every team needs a leader, and right now that team is all who regularly
contribute to DebOps and its leader is you. I'm not sure if creating
another team within said team would change that much. More involvement
in testing/approving PRs however might.
Actually, I was wondering if creating a new mailing list for such
people interested in maintaining the project and giving it direction
would be a good idea. Say, 'debops-devel'? I'm sure that not every
user would be interested in such discussions, this would let them
have a choice if they want to keep tabs on them or not.
I'm okay with having this on debops-users, it's not like there is too
much list traffic here. We can always move it to another list if things
really take off.
> Do you feel like the project is (heavily) dependent on you?
It is, a bit by my own design, and this probably creates a bottleneck
in the speed of changes and new features impelemnted in DebOps. I'm
not sure what pace of development would be expected from such a
project; Debian Stable releases happen about every two years and it
is considered very slow by Linux community standards. I implemented a
four stable releases per year model which kind of fell apart a bit
over time, totally by my own hand. I need to better take care of it
in the future, but if this model proves unsustainable I'm not against
changing it. I guess more discussions about it will be needed.
I'm all in favour of the "It's done when it's done" mentality. This
of
course only works when there are enought people working on getting it
done. So if you ask me, the most important drivers of a new release are
the people who work on it, and the system they use to make the time
they assign to the project as worthwhile as possible.
That last part can translate to project management methods that I don't
want to spend too much time on discussing, but keeping something like a
todo list up-to-date could be considered essential.
> Is your involvement causing you any sort of stress?
Hmm, maybe a bit. I definitely feel guilty of not handling incoming
pull requests or issues on GitHub in a timely manner, I will have to
work on that.
I think we should help you here, you're obviously too busy with other
things. I guess everyone is in some way, but reviewing PRs and
prioritizing them sounds like easy money to me. You can of course still
leave the final merge decision up to yourself, but more public
discussion on PRs can't do much harm. And you'll be able to make a more
informed decision.
Other than this, the scope of what the project does really broadened
over the years and the development pace slowed down as a result. On
each stable release I try to set a few goals for myself for the next
one, some of them are made, some not. I definitely try to focus on
lower-level stuff, roles that are used in many different places
(think firewall, LDAP, various databases) rather than specific user-
facing applications so that development efforts can be multiplied
over multiple roles at a time. On top of all that is all of the non-
Ansible related things and management of the project itself (DNS,
various services used for DebOps development, etc.). Of course on top
of that is my regular workplace stuff which sometimes needs to take
precedence.
I'm jealous of the amount of work-hour time you can (seamingly) put
into this project. Maybe I should allocate some block of hours every
week to improving DebOps roles. I've been doing it between other things
until now, and always in relation to some automation problem I've been
having at work. The DHCP role would be a great starter.
The improvements to lower-level roles indeed sound like a great way to
multiply your efforts.
It's definitely good to hear that your employer is aware that
it's
not all in your fingertips, but you use open projects for your work.
I think that's an excellent example to follow.
Haha, it's like I said. My employer writes a lot of open source
software that in itself leans on other open source software. It's the
open source redistribute-athon :)
Currently most of the real-time communications within the project is
done on IRC, but there's no logging involved and the format of an IRC
conversation is probably not very good for some things. More focus on
the mailing lists would probably benefit most current and future
DebOps users, especially that we now have proper search UI in Mailman
3.
It would benefit me a lot, I'm not really the IRC type. I know how to
idle there but am almost never actively online.
An online meetup is an interesting idea, this would definitely help
put some faces to the nicks. I'm up for it if anyone else wants to
meet up. We can arrange a time and method later via the mailing list;
it should be easy to do using some of the many online services.
Let me propose a proposal. Let's propose next Wednesday at 18:00 *UTC*.
Let's also keep the expectation that it will be relatively short (~30
mins) so that many people will join. We can do a little meet&greet and
discuss plans. I can take the lead if you want, or you can do it
yourself.
What I would be interested in is access to more varied
infrastructure
for development. At the moment I have limited resources at home, and
development boxes that are available to me at work are extremely slow
(How slow? the common playbook on my home workstation takes about 4
minutes to finish. On the development server at work? 35 minutes.).
Due to what I have available, I'm doing my development using LXC
containers which introduces a bias in what I'm currently working on.
Some things are not possible using containers exclusively, for
example NFS infrastructure needs at least VM-level setup since you
cannot mount filesystems in unprivileged containers. A VM management
platform like OpenNebula requires either hardware-level access or
virtualization nesting which introduces speed penality as well (that
for example would grind my development boxes to a halt). It would be
interesting to have access to some boxes and a subnet, of course if
latency over SSH is reasonable.
Well that's something I can help out with. You'll get a 2 vCPU VM with
4 GB RAM, 50 GB storage and unfiltered Internet access with a public
/29 and /64. Everything is easily expandable if need be. We're just
waiting on the (likely overworked) colocation provider to assign the
subnets. I like you Maciej, but I don't want to put you in our
business-critical production VLAN just yet ;)
Publicity... I really suck at it. And then there's the mentality
on
the Net (for example Reddit) where advertising your own work can be
considered akin to shilling... So I just don't do it. But if you
think that DebOps is worth it, and you have some good ideas on how to
do it - be my guest. :-) Let me know if you need some pointers about
various aspects of the project. Just be aware that this will bring
more users to DebOps which will result in more issue reports and pull
requests which might be good or bad - who knows. :)
That mentality can really suck sometimes. But hey, so far I've been
boasting around how successful one can be automating their
infrastructure with DebOps, so maybe that will help a little. And then
there's that blog post that I'm writing anyway. My employer has also
been boasting around about the automation tech we use, so I can imagine
that at least some people around us are very interested right now :)
Having more users is the first step to having more contributors!
Cheers,
Imre