Moss Format Defined
The core team have been hard at work lately implementing the Moss package manager. We now have an initial version of the binary format that we’re happy with, so we thought we’d share a progress update with you.
Explaining the format
Briefly, the binary container format consists of 4 payloads:
- Meta (Information on the package)
- Content (a binary blob containing all files)
- Index (indices to files within the binary blob)
- Layout (How to apply the files to disk)
Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload
both compressed and uncompressed, the compression algorithm used (
zlib supported) as well as the type and
version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order).
Internally the representation of a Payload is defined as a 32-byte struct:
@autoEndian uint64_t length = 0; /* 8 bytes */ @autoEndian uint64_t size = 0; /* 8 bytes */ ubyte crc64 = 0; /* CRC64-ISO */ @autoEndian uint32_t numRecords = 0; /* 4 bytes */ @autoEndian uint16_t payloadVersion = 0; /* 2 bytes */ PayloadType type = PayloadType.Unknown; /* 1 byte */ PayloadCompression compression = PayloadCompression.Unknown; /* 1 byte */
We merge all unique files in a package rootfs into the
Content payload, and compress that using zstd. The offsets to
each unique file (i.e. the sha256sum) are stored within the
Index payload, allowing us to extract relevant portions
from the “megablob” using
These files will become part of the system hash store, allowing another level of deduplication between all system
packages. Finally, we use the
Layout payload to apply the layout of the package into a transactional rootfs.
This will define paths, such as
/usr/bin/nano, along with permissions, types, etc. All regular files will actually
be created as hard links from the hash store, allowing deduplication and snapshots.
Meta payload consists of a number of records, each with strongly defined types (such as
with the tag, i.e.
Summary. The entire format is binary to ensure greater resilience and a more compact
representation. For example, each metadata key is only 8 bytes.
@autoEndian uint32_t length; /** 4 bytes per record length*/ @autoEndian RecordTag tag; /** 2 bytes for the tag */ RecordType type; /** 1 byte for the type */ ubyte padding = 0;
TLDR that for me…
Binary format that is self deduplicating at several layers, permitting fast transactional operations.
Before we work any more on the binary format, we now need to pivot to the source format. Our immediate goal is to now
have moss actually build packages from source, with resulting
.stone packages. Once this step is complete we can
work on installation, upgrades, repositories, etc, and race to becoming a self hosting distribution.
Note, the format may still change before it goes into production, as we encounter more cases for optimisation or improvement.