tinc vpn with brigde on all nodes
by Jan Kowalsky
Hi all,
trying to understand the tinc role I wondering if there is an
possibility to set up mesh network where all tinc tap adapters are part
of a bridge (br2) on their hosts.
The goal is to have a server cluster where vms on different host can
share one bridge which is connected through tinc.
The configuration below results in a situation where none of the hosts
have a "ConnectTo=" variable in tinc.conf.
If I use the "link_type: 'dynamic'" we have these variables - but the
tap-mesh0 interface itselfs tries to get address via dhcp - which is not
what I want.
I also wondering if it is intended that taking the example inventory
(https://docs.debops.org/en/master/ansible/roles/tinc/getting-started.html...
with one host with dns/dhcp) there is no ConnectTo neigher on the host
which provides DNS/DHCP. Doesn't this lead to an setup which is not a
real mesh network?
This example configuration (one dhcp host - other nodes where tap-mesh0
gets address through dhcp) works for me (but only with bridge on host1).
But the host entries in mesh0/hosts/hostXY have the hostname configured
with the "br2" subdomain - like host1.br2.example.org which is the
name-resolution of dnsmasq on host1. Does this makes sense? Why not
address all hosts through there fqdn on the primary interface? Then we
don't get any problems with static bridge configuration without dhcp and
working name resolution via tinc.
Additionally it seem's not so easy to find the right order for getting
up bridges and the tap interfaces:
Even if fix the problems above resulting in a setup where on host has
tap interface on br2 with dhcp server and two other host with tinc via
bridge configuraion and dhcp: after reboot I have first to restart tinc
before it's possible to get an ip address through dhclient.
One of my test configs:
Host 1:
tinc__host_networks:
'mesh0':
bridge: 'br2'
ifupdown__interface_layout: 'dynamic'
ifupdown__host_interfaces:
'br2':
type: 'bridge'
inet: 'static'
nat: True
addresses:
- '10.0.0.11/24'
Host 2:
tinc__host_networks:
'mesh0':
bridge: 'br2'
ifupdown__interface_layout: 'dynamic'
ifupdown__host_interfaces:
'br2':
type: 'bridge'
inet: 'static'
nat: True
addresses:
- '10.0.0.12/24'
Host 3:
tinc__host_networks:
'mesh0':
bridge: 'br2'
ifupdown__interface_layout: 'dynamic'
ifupdown__host_interfaces:
'br2':
type: 'bridge'
inet: 'static'
nat: True
addresses:
- '10.0.0.12/24'
Best regards
Jan
3 years, 9 months
New stable DebOps release: v2.1.0
by Maciej Delmanowski
Hello everyone,
It's time for a new DebOps release, this time with a sliiight delay due
to various factors. But it's here!
New DebOps release, v2.1.0
--------------------------
You can find the new version of DebOps on:
GitHub: https://github.com/debops/debops/releases/tag/v2.1.0
PyPI: https://pypi.python.org/pypi/debops/2.1.0
Galaxy: https://galaxy.ansible.com/debops/debops/
You can upgrade the Python-based installation by running the command:
pip3 install --upgrade debops
Installation instructions can be found here:
https://docs.debops.org/en/stable-2.1/introduction/install.html
The brief Changelog can also be found on the documentation page:
https://docs.debops.org/en/stable-2.1/news/changelog.html
Complete, detailed changelog can be viewed using the 'git log' command.
You can use the 'git log --no-merges' command to skip the "boring" merge
commits.
The DebOps documentation has a separate page which details important
changes from previous release in the Ansible inventory or on the remote
hosts which you might need to perform manually:
https://docs.debops.org/en/stable-2.1/news/upgrades.html
The Python packages available on PyPI, as well as the tarballs available
on GitHub are signed with my GPG key. You can get it from the OpenPGP
keyserver network using the command:
gpg --keyserver hkp://pool.sks-keyservers.net \
--recv-keys 27067A91D620EE91D50309D92DCCF53E9BC74BEC
This release was meant to be done in April 2020. Unfortunately for
various reasons it didn't happen, sorry about that. Due to the delay,
the release schedule will be changed a bit to accomodate the
mid-development cycle release; the next one will be in October 2020.
The v2.1.0 development cycle finished with 490 commits (not counting
merges) which is pretty average for the project. Here's a breakdown of
committers this time around:
340 Maciej Delmanowski
81 Robin Schneider
19 Pedro Luis López Sánchez
10 Imre Jonk
7 Gabriel Lewertowski
6 Rainer 'rei' Schuth
6 Tasos Alvas
5 Hendrik Visage
2 Alin Alexandru
1 Ant Somers
1 Bao Nguyen
1 Hanno - Felix Wagner
1 Jean Sorgemoel
1 Jonathan Struebel
1 Jérémy Gardais
1 Korbinian Preisler
1 Patryk Ściborek
1 Paulo da Silva
1 Sandro Jäckel
1 Serge Victor
1 Steffen Lindner
1 Thiago Talma
1 Thomas Danielsson
Thanks to everyone involved, and see you in the commits.
Goals reached since previous release
------------------------------------
As usual, only part of the plans for the next DebOps release[1] was
finished before the next one. Let's see what goals were reached:
- New DebOps roles - 7 new roles were added to DebOps (6 if you count
InfluxDB client/server together). That's less than I hoped, but most
of that is on me not doing reviews of the pull requests. I'll try to
do better next time.
- The "wrapper roles" for firewall, NTP and webserver services didn't
make the cut eiher, but I still plan to introduce that concept in the
future. Stay tuned.
- Refresh of the Mailman role - done! You probably are reading this on
the new Mailman 3 instance that manages DebOps mailing lists.
Unfortunately, changes in the application broke some of the mailing
list URLs referenced in older e-mail messages, but hopefully the new
search functionality can mitigate this.
- Support for ELK stack and logging - partially done. I didn't get
around to writing the Logstash role, but @PedroLuisLopez started
working on one[2]. If you are interested in that service, check his PR
out and leave a review.
What I did do is add the 'journald' Ansible role and refresh the
'rsyslog' role, so at least the start of the logging pipeline should
be modernized.
[1]: https://lists.debops.org/hyperkitty/list/debops-users@lists.debops.org/th...
[2]: https://github.com/debops/debops/pull/1370
The 'stable-1.0' release is retired
-----------------------------------
DebOps v1.0.0 was released on 22th May 2019 and had 18 patch releases
with about ~390 commits; most of these were backports from later
releases. Major highlights of this release included introduction of
a redesigned LDAP support project-wide, role for Docker Registry that
integrated with GitLab and removal of the 'bootstrap' role - its
functionality was moved to various other roles. It was also the first
release published on the new Ansible Galaxy as an Ansible Collection.
There will be no new updates to the 'stable-1.0' branch. Existing users
are encouraged to upgrade to newer DebOps releases, and to read the
upgrade notes for information about needed changes.
The next DebOps release to be retired will be 'stable-1.1', on 25th
August 2020.
Additionally, the 'stable-1.1' and 'stable-1.2' releases will not be
uploaded to Ansible Galaxy anymore due to too many changes needed to
make them compatible with Galaxy requirements. These branches will be
retired in the next few months, so that shouldn't be a problem; users of
the old branches should switch to the 'stable-2.0' branch or newer.
New release highlights
----------------------
There were many changes in the project since the v2.0.0 release, many of
them were backported to the previous releases as well. Here are some
highlights from the Changelog:
- Starting with the v2.1.0 release, the DebOps project is now compliant
with the REUSE Specification[3] which defines how Open Source projects
should maintain Copyright and License information. All files in the
DebOps repository have been audited, copyright information and
licensing is up to date and the compliance is checked[4] on Travis-CI
before any pull requests are merged.
- The annoying "Could not match supplied host pattern" warning messages
during Ansible playbook execution will be suppressed via Ansible
configuration options. Existing deployments have to be modified
manually to handle this, though.
- Support for Ansible Collections is much better now, bugs with wrong
collection references in Ansible plugins have been ironed out. The
DebOps Collection has been split into separate smaller collections due
to the artifact size limits imposed by Ansible Galaxy service. Only
the main collection ('debops.debops') is mandatory, others have only
additional roles. Installation of role collections should be handled
by the 'ansible-galaxy' tool using collection dependencies.
- Ansible now handles undefined nested dictionaries cleanly so
most variable definitions like:
var: '{{ ansible_local.core.admin_users
if (ansible_local|d() and ansible_local.core|d() and
ansible_local.core.admin_users|d())
else [] }}'
have been reduced to:
var: '{{ ansible_local.core.admin_users | d([]) }}'
Thanks go to @ypid for handling the brunt of the work updating these!
[3]: https://reuse.software/
[4]: https://api.reuse.software/info/github.com/debops/debops
Third party code vendoring and Ansible Collections
--------------------------------------------------
During previous release I mentioned an issue with use of the third-party
code in DebOps via Ansible Collections or some kind of vendoring
solution. After thinking about this issue for some time, I came to the
conclusion that DebOps needs to be fully converted to an Ansible
Collection model instead of modifying the existing codebase in the
collection build script to conform to the requirements of Collections.
With this approach, DebOps will be fully compatible with other Ansible
Collections in all three usage models (git monorepo, Python package,
Ansible Collection). DebOps roles will be able to reference roles,
modules and other Ansible plugins in third-party Ansible Collections.
At the moment the Collections don't have any kind of GPG-based
verification possibility, I'm not sure how to solve that effectively.
This of course will require some changes in the monorepo like
introduction of 'ansible/collections/ansible_collections/debops/debops'
symlink that points to the 'ansible/' directory, and the file structure
in the Python package will have to be rearranged, but these are minor
details. The 'ansible_plugins' role will also become unnecessary since
Collections can natively resolve custom plugins and modules when they
are specified using FQCN names.
Another thing that will have to be done before the changes are
implemented is migration of all 'debops-contrib.*' roles to the main
namespace. This has to happen to allow them to be used in Ansible
Collection model.
The breaking change will happen in the playbooks and roles maintained by
DebOps users. The playbooks and roles need to include the 'collections:'
keyword to tell Ansible where to look for roles and plugins; in the roles
themselves references to filter, lookup and other plugins will have to be
modified as well. The changes to roles needed to make them
collection-compatible will break the "legacy" usage as standalone Ansible
roles, so the change needs to happen everywhere at once, aka "flag day"[5].
This will be a major change, so the next DebOps release will be v3.0.0. Longer
release cycle this time will hopefully help with development and release on
time. Since the change will happen on the 'master' branch at some point, if
you don't want to be affected by it unexpectedly, you might consider sticking
to the 'stable-2.1' branch for the next development cycle.
[5]: https://en.wikipedia.org/wiki/Flag_day_(computing)
Redesign of the 'debops*' scripts and project directories
---------------------------------------------------------
The 'debops*' scripts will also have to be updated to support the new
Ansible Container-based model. This is a good opportunity to clean up
and redesign them, which I plan to do in the coming months. Some of the
things I would like to implement:
- change all 'debops-*' commands to be subcommands and implemented in
Python instead of bash where necessary
- add support for multiple Ansible inventories in one "project"
directory with one inventory being a primary one. Inventories can then
be treated as "views", where the primary inventory manages the system
itself, additional inventories are configured to use unprivileged user
accounts to deploy applications, etc.
- implement better management subcommands on the Ansible Controller to
maintain project directories. Since DebOps is used from different OS
platforms (Linux/macOS/WSL/FreeBSD), keeping it cross-platform should
be easy with Python.
- add support for a Bridge CA to allow different environments to talk to
each other securely over TLS. This would also allow for easier
initialization and managemnt of the internal Certificate Authorities
on the Ansible Controller without an awkward setup by running the
'pki' role against a host.
- native support for project directories stored in git repositories,
with separate git repository for each Ansible inventory. This would
enable sharing the projects (we really need a better name for that
concept) between users in a distributed environment with tracked and
synced changes. The 'debops' scripts could automatically commit
changes, check remote repositories for updates, and so on.
- add support for runtime synchronization and communication between
'debops' execution processes via external services like RabbitMQ,
MQTT, and similar. This would simplify coordination between system
administrators and other users in distributed setups.
It will be a good opportunity to learn some good Python practices for
me. For inspiration I plan to look to AWX project as well as FreeIPA,
although focused on usage via CLI and more hands-on approach. DebOps is
good in bootstrapping new environments, and I would like to keep it that
way.
If you have any more interesting ideas that could be added to the
scripts after the refactor is complete, let me know.
Plans for the next development cycle
------------------------------------
Redesign of the 'debops' scripts and switch to the Ansible Collection
model will probably take significant amount of time in the next
development cycle, so I don't plan to introduce large changes in the
existing DebOps roles. However what I want to do is add support for the
'firewalld' service and improve firewall configuration throught the
project, so hopefully there will be time to implement this before
October.
Till the next time,
Maciej
3 years, 10 months
A few roles that might be of interest
by Imre Jonk
Hi all,
The company I work for just open-sourced these roles that were written
for use with DebOps:
- debops.simplesamlphp
Manages SimpleSAMLphp as a SAML iDP. Installs
simplesamlphp-module-webauthn by default, which adds WebAuthn 2FA.
- debops.selfservicepassword
Manages Self Service Password, a web interface for changing one's LDAP
password.
- debops.openvpn
Manages OpenVPN server for client-server use. Authenticates clients
against LDAP and integrates with YK-VAL to provide Yubikey TOTP 2FA.
- debops.matomo
Manages Matomo analytics platform.
- debops.dhcpd
Manages ISC DHCP server. Essentially a rewrite of the existing
debops.dhcpd role with support for DHCPv6, but it's still missing some
features.
- debops.dnsui
Manages Opera DNS UI, a web-based LDAP-authenticated DNS management
system for use with PowerDNS authoritative nameserver.
- debops.powerdns_auth
Manages PowerDNS authoritative nameserver.
We use all these roles in production. A lot of documentation is still
missing, I hope to be able to add that soon so the roles can be
integrated into DebOps mainline. The roles are licensed under
GPL-3.0-or-later and can be found here:
https://gitlab.com/ciphermail?filter=debops
Suggestions for improvements are more than welcome.
Imre
3 years, 10 months
Migration to Mailman 3 (v2)
by Maciej Delmanowski
Hello everyone,
I've just completed migration of the DebOps mailing list server from the old
Mailman 2.1.x version to Mailman 3.x installed on Debian Buster - all with
DebOps of course. The migration was done by setting up a new VM, therefore IP
address of the server has changed, which might impact e-mail delivery. The new
service also uses DomainKeys to sign outgoing mail messages, which also can
impact message filtering on the recipient hosts.
This message is a final test to see if everything was set up successfully
- and indeed there was a problem with permissions on the server. Let's hope
that this time the mail messages will be sent.
If you want, you can check the new web interface and archives on
https://lists.debops.org/. Let me know if there are any issues with the
service.
Now back to our regularly scheduled programming,
Maciej
3 years, 10 months
Migration to Mailman 3
by Maciej Delmanowski
Hello everyone,
I've just completed migration of the DebOps mailing list server from the old
Mailman 2.1.x version to Mailman 3.x installed on Debian Buster - all with
DebOps of course. The migration was done by setting up a new VM, therefore IP
address of the server has changed, which might impact e-mail delivery. The new
service also uses DomainKeys to sign outgoing mail messages, which also can
impact message filtering on the recipient hosts.
This message is a final test to see if everything was set up successfully. If
you want, you can check the new web interface and archives on
https://lists.debops.org/. Let me know if there are any issues with the
service.
Now back to our regularly scheduled programming,
Maciej
3 years, 10 months
Monitoring systems with DebOps
by Nicolas Quiniou-Briand
Hello,
I will deploy a monitoring solution on a host managed by DebOps and I
would like to:
- get your feedback on monitoring solutions currently part of DebOps
(like monit, Icinga and others [1])
- know futur of monitoring in DebOps: is there any plans to support new
softwares ?
Thanks a lot.
[1] https://docs.debops.org/en/master/ansible/role-index.html#monitoring
--
Nicolas Quiniou-Briand
Messagerie instantanée (Jabber) : nqb(a)azyx.fr
Tél : 06 75 84 56 74
3 years, 10 months
debops install methods, updates and rolling release
by Jan Kowalsky
Hi to all,
today, after installing debops via the debops.debops role on an remote
ansible controller I realized that I was always confused about the
debops mono repo with rolling release vs the pip install command.
What I (hopefully) understood:
* when installing debops via pip everything is under
./local/lib/python3.7/dist-packages
* after debops-update there will be an additional debops repo in
./local/share/debops/debops/
The latter is on git branch master - while the repo from pip is on the
release tag - but once I have an repo in local/share/debops it will have
precedence over the first according to file path order in .debops.cfg.
Right?
What I did all the time working with debops, was ignoring the pip
installation and used git pull && git checkout <Release-Tag> to update
my debops to an stable release.
Now I realized that the debops.debops role always automatically cloned
the monorepo to /usr/local/share/debops/debops - in the master branch.
No I wondering:
* is it a good practice at all just to use git checkout inside the
monorepo - what I did all the years to get an specific release?
* does the debops.debops role always lead to an debops version on
branch master or is there an possibility to configure an release?
And one last question: what purpose has the [debops_recursively] host group?
Thanks and kind regards
Jan
3 years, 10 months
