Ready, Set, Go
While the blog has been quiet, the git commits have been flowing! The core pieces of our infrastructure are nearing the testing phase - a complete workflow for developers to submit and build packages and for them to become available in the Serpent OS repo. At this point we can get many more contributors involved in packaging while circling back to make it more automated and efficient for them.
By respecting the time of contributors, we can get more done in the same time, utilise and manage more contributions to reduce the workload on everyone. This also makes Serpent OS an attractive platform for new contributors and enables more users to get involved by making packaging so easy.
TL:DR - With the infrastructure and tooling almost in place, the bringup of Serpent OS begins
The Long Road Reaps Rewards
What we have done is worked on getting all the tools and infrastructure in place before trying to release anything installable to users. Experience has shown that large refactors or complete rebuilds of the tools are difficult to pursue and take years to eventuate (if at all). This means that efforts can now go towards making the distribution you've been waiting for without having issues scaling or tools that need a rewrite due to hurting the experience of contributors.
Going forward users and developers will see tangible benefits from the work put into Serpent OS without significant time spent on the hidden parts of the distribution (which are essential but time consuming). This is not to say that they are feature complete, but that the focus can be on making visible improvements to people's Serpent experience.
From a fundraising point of view, this has meant it has taken longer to get a usable ISO into your hands...but the benefits will be seen for years to come.
summit - Managing the Distribution
At a high level,
summit sits in the middle of everything, communicates with our other tools
and provides an overview of the project and what's happening within it via dashboards. When a build is generated,
summit fetches the build information, determines the optimal build order, finds a builder to create the package and
vessel know when it's ready to enter the main repository. This is all backed by authentication to ensure
only authorised users and builders are used to maintain security.
Our build files are backed by git repositories on GitHub, where
summit brings the important information much closer to
developers. Our git repositories are organised into logical groups representing the delegation of responsibilities. For
example, we can add many more developers to our
plasma group and enable them to push builds directly where we may not
want them to modify the toolchain or kernel without a second review.
summit also parses information from git to display in an easy to consume format. Here we see an example of the
metadata page of the
avalanche - Builder as a Service
avalanche runs as a service ready to accept builds from
summit. It is low overhead and monitors the state of the
machine via a dashboard so if the builder were to run out of disk space or stall it will be easily seen by the
avalanche owner. This makes it very easy for people to spin up a builder and will eventually be utilised for people
who lack powerful hardware to test builds on before submitting as official packages. Note that distribution packages
will only be built on the most trusted builders and test builds will be made to ensure the correctness of all
avalanche builders is maintained.
There are many use cases for
avalanche, where we can partition system resources via
systemd-nspawn instances to
provide multiple builders on one host. Many builds only utilise a few cores, so it becomes more efficient to run
multiple builders concurrently. As builds are isolated from the host,
avalanche instances can easily be run on
bare metal or in a container/VM if preferred.
Once a build is completed,
avalanche is responsible for reporting completion to
summit then instructs
vessel to fetch the build artefacts from the relevant
This design decision allows
summit to schedule a new build to the
avalanche instance as soon as it has reported
completion of the prior build, thus allowing for maximum use of both upload and download bandwith for each
instance -- and as a direct consequence, higher overall build throughput across the entire set of builders.
vessel - Making Packages Available to Users
vessel is a fairly simple (but very important) part of the build infrastructure. After fetching a package from a
vessel will then update its index with the newly minted package, thus making the package available in
our official repository for installation.
vessel will be expanded to support versioned indexes (making each update a new release) and branches to
enable testing new features complete with rollback that
moss supports across all updates.
Enabling Faster Packages!
Feature work has started on adding ISA levels to enable support for adding
x86-64-v3x packages. This can provide extra
performance and reduced power use for supported CPUs (this denotes the
x86-64-v3 psABI with some extra extensions
that your processor will likely also include). As we aren't relying on the
glibc hwcaps feature, we have the ability
to adjust the featureset to the benefit of our users. Once we have a
x86-64-v4 builder available, we can also look at
providing an extra layer for packages that heavily utilise math. This will be the result of actual benchmarks to
indicate which packages provide additional performance from their newer CPU features.
With the infrastructure already supporting multiple builders (and an optimised toolchain to minimise build time), adding more architectures will not pose any capacity issues.
Finally, a note on fundraising
Recently we switched away from OpenCollective for our fundraising to GitHub Sponsors, as the platform offers a zero-fees sponsorship. Being an independent project - every penny really does count! However, we did lose an element of transparency, so I'd like to provide some initial data:
- October 2022: £3,003.70
- November 2022: £540.58
At some point this website will incorporate some more up-to-date information on sponsorship information to facilitate a more transparent approach. Recently I've taken some additional steps towards this, including opening a (Revolut) bank account specifically for Serpent OS funds + expenses.
I'd like to thank everyone supporting the project - the time for us to reward you has now come. Intermittent funding and the cost of living crisis has had a huge impact on the timescale of the project to date, however we're not shying away from the challenge. Rather, we've cracked on with implementing the proto-infrastructure and will be deploying it within days.
Let's do this.
You can discuss this post over on our forums