Hello everyone, Yesterday the number of pending DebOps pull requests on GitHub reached 0. Therefore, today I merged the current code from most of the DebOps roles and some other repositories. The new repository is available at: https://github.com/debops/debops.git The repository contains code from the 'master' branch of most of the DebOps roles as of 2017-10-21. This repository should now be considered as a base DebOps repository, and all changes to the existing roles, or any new roles and other code, should be directed to it. Existing DebOps repositories ('debops-tools', Ansible roles, documentation, etc.) haven't been changed and they are still available. The 'debops' scripts installed from PyPI will still use the old repositories to install DebOps; the same goes with installing individual roles via Ansible Galaxy. How the repository merge was done --------------------------------- For the curious about the migration process from multiple git repositories to a single "monorepo": the main obstacle was the same directory and file names in Ansible role repositories (ie. 'defaults/main.yml', etc.). The easy solution would be to just copy the files directly to the new repository, but I wanted to preserve the git commits in the new monorepo, for posterity and history. First, I downloaded all of the existing DebOps roles via 'debops-update' script to a new directory to have a clean workspace. Then in each git repository, I performed the commands (an example from the 'debops.users' role); mkdir -p ansible/roles/debops.users git mv CHANGES.rst COPYRIGHT defaults docs .gitignore LICENSE meta \ README.md tasks templates .travis.yml UPGRADE.rst \ ansible/roles/debops.users git commit -m 'Move role code to a subdirectory' This ensured that the code of each Ansible role and other DebOps repositories (debops-playbooks, debops-policy, debops-api, debops-tools) would be placed in a separate directory after the merge in the monorepo. After the code in all of the relevant repositories has been moved out of the way, I switched to the main monorepo and started merging the repositories one by one: git remote add -f ansible-users ../roles/debops.users git checkout -b role-users ansible-users/master git checkout master git merge --no-ff role-users With this, the existing git commits in the separate repositories have been merged directly into the 'master' branch of the new monorepo. There's no submodules, no subdirs, all of the code is directly in this repository. The git tags were an issue after the merge - since git allows only one unique tag with the same name, and DebOps roles used the same names in tags (v0.1.0, etc.), merging them over was impossible - renaming would just result in a new tag. Therefore I decided to remove all of the existing tags and start from scratch. Since the last tagged version was v0.5.1, I tagged the current code as v0.6.0 to avoid collision with the old repositories. Hopefully this will quickly go to v1.0.0, but we will see how it goes. :-) The size of the '~/.local/share/debops' directory on my workstation sits now at 168940 kilobytes (165M) as reported by 'du'; the new repository is 40556 kilobytes (40M). There are 9314 commits at the moment, but that's only part of th whole project - some repositories like 'test-suite' or 'examples' hadn't been merged. You can check out some of the statistics of the repository on GitHub. I omitted some of the existing DebOps roles during the merge: - 'debops.backporter' was due to be deprecated. The role was created before Debian Jessie has been a Stable release, to aid with backporting Ruby 2.1 packages to Debian Wheezy, required at the time by GitLab. Since then, Jessie has become Stable and is now Oldstable, GitLab is installable directly on current Debian Stretch, and concept of automatically backporting Debian packages from Testing distribution to Stable has been dropped in DebOps. - 'debops.gitlab_ci' role was created when GitLab CI was a separate application. Since then, GitLab CI has been merged with the main GitLab application and the role was deprecated. - 'debops.gitlab_ci_runner' was the remote Runner for old version of GitLab CI. It was replaced by GitLab CI Multi Runner some time ago, and the role was deprecated in favor of 'debops.gitlab_runner' which supports the new version. - 'debops.kvm' role was used to set up a KVM environment on a host. This functionality has been moved to the 'debops.libvirtd' role and the KVM role was deprecated. - 'debops.monkeysphere' role was in the project almost from the beginning and was meant to aid in configuration of The Monkeysphere Project on the managed hosts. I haven't used it in a long time, and I hadn't heard about anyone else using it either, so I decided that this is as good moment as any to deprecate and remove it. The support for Monkeysphere might return someday. - 'debops.mysql' role was some time ago deprecated and replaced by the 'debops.mariadb_server' and 'debops.mariadb' roles. - 'debops.php5' role has been deprecated in favor of the 'debops.php' role. Apart from the existing DebOps roles located in the 'ansible/roles/' directory, in the main repository you can currently find: - 'lib/debops-api' - contents of the https://github.com/debops/debops-api/ repository. - 'lib/debops-policy' - contents of the https://github.com/debops/debops-policy/ repository. - 'lib/debops-tools' - contents of the https://github.com/debops/debops-tools repository. - 'ansible/playbooks' - contents of the https://github.com/debops/debops-playbooks repository. The project playbooks are currently located in an awkward 'ansible/playbooks/playbooks/' subdirectory because I wanted to preserve the existing git commits of that repository. They will be moved to the parent subdirectory at some point. Current state of the project ---------------------------- At the moment, the new repository does not have any Travis-CI or GitLab tests. Because of that I don't want to rush with more involved code changes just yet. I'm looking into writing tests for specific DebOps roles and their combinations using GitLab CI, based either on Vagrant-LXC or Vagrant-libvirt test environments. Tests based on Docker are insufficient for DebOps, since various playbooks and roles can manage multiple services at once - Docker traiditonally uses one container per service which clashes with the DebOps-managed environment. Quick evaluation of Vagrant-LXC based test environment is promising, apart from the more low-level roles like management of network interfaces since 'systemd' inside LXC containers doesn't allow for bringing the network interfaces correctly in Debian Stretch; due to that a test environment based on Vagrant-libvirt and a KVM backend seems to be the best fit for these roles. As for the Travis-CI tests, I want to create a general suite of tests which will check the project's documentation and validity of the Ansible code without executing the playbooks and roles directly. At the moment the main project repository cannot be installed or updated using the 'debops-update' command, and is unusable via the 'debops' command without some manual intervention. If you want to base your environment on the roles in the new repository, easiest way to do it would be to symlink the 'ansible/roles/' and 'ansible/playbooks/playbooks/' directories from the main repository to the respective 'ansible/roles/ and 'ansible/playbooks/' directories in a given DebOps project directory. The next steps -------------- There are currently 281 open issues across the old DebOps roles. They will be resolved over time by fixes in the main DebOps repository. Please create any new issues or pull request against the 'debops/debops' repository on GitHub and not the old, separate role repositories. Cleanup of the old codebase is a good and easy way to start. Removal of the redundant LICENSE, .travis.yml files and the like. Perhaps even the README files in the role directories can go as well, they will be preserved in the git history, and they aren't exposed as much in the new repository. The documentation from role directories, playbook directory and separate policy directory should be moved to the the new 'docs/' directory, with a Sphinx build environment. When this is ready, the new documentation will be available at https://docs.debops.org/ as usual, via ReadTheDocs. A few people on IRC expressed interest in updating the 'debops-tools' scripts. It's a good opportunity to write them from scratch with a new command line interface based on subcommands. I imagine that it will take a while before this is ready though. For now, you can just clone the repository directly to install it. I plan to look into GitLab-CI tests and prepare the tests for as many roles as I can using the Vagrant-LXC environment, so that updates can be easily checked. I don't plan to create a public GitLab instance for that yet, instead you will be able to deploy your own test environment based on DebOps. At the same time, I plan to remove the example playbooks from the role 'docs/playbooks/' directories and use the main project playbooks as examples in the documentation (if possible, not sure yet), and for role testing. I want to reach out to a few Debian Developers interested in packaging DebOps in Debian. With the new repository it should be much easier to do than before. Hopefully with their help, a 'debian/' directory can be added in the repository so that users can build their own '.deb' DebOps packages and install the project that way. Well, that's it for now. Let me know if you have any questions or find issues with the new repository. Cheers, Maciej