SDD 0006 - Commodore Configuration Catalog Compilation

Kapitan is a tool to render templates in different templating languages (currently Jsonnet, and Jinja2 are fully supported, Kadet and Helm are in alpha status. Kadet is a python-based, Jinja-inspired templating language for YAML). Kapitan works with components and an inventory. Kapitan components are collections of templates which should be rendered. The Kapitan inventory is similar to Puppet’s hieradata and consists of "classes" and "targets." A Kapitan target describes a unit which will be compiled by Kapitan (for example all the manifests required to deploy an application on a cluster). Each Kapitan component also requires a "component class" which describes how the component should be rendered. This includes listing which inputs to render, and where to save the outputs, the input format (on of the supported templating languages), and the output format (for example YAML or JSON). Additionally, Kapitan itself can download dependencies of a component, for example the helm chart if using helm as the templating language.

For the catalog compilation engine, we define all of the platform configuration in a single Kapitan target named "cluster," which is constructed from data provided by the Lieutenant API. The target pulls in a generic inventory class for global configuration (named global.common), cloud-specific configuration (for example class global.cloud.aws) and cluster distribution specific configuration (for example global.distribution.rancher). Additionally the target also pulls in a class for the cluster itself (for example class thecustomer.thecluster). Finally, the compilation engine traverses the inventory to gather all the components which are referenced, downloads them, and ensures that a class components.thecomponentname and defaults.thecomponentname (to define component defaults) is available in the Kapitan inventory. See Component Repositories below for a more detailed description of how we manage Kapitan components.

During compilation, we enable Kapitan’s dependency fetching to allow components to use the dependency mechanism to download their internal dependencies. We configure Kapitan’s search path to include the dependencies directory. This allows components to omit the leading dependencies/ in their compilation descriptions and enables the framework to provide extra Jsonnet (or other templating) libraries that are available to all components. The compilation engine downloads a list of libraries (defined in the inventory) and makes them available under dependencies/lib which in turn is visible under lib from the perspective of components.

The engine also provides a postprocessing step which runs after Kapitan. This postprocessing step can modify the rendered output, for example to add an explicit namespace into a rendered Helm chart, as helm doesn’t support adding namespace information when using helm template . The postprocessing step works similar to Kapitan’s compile step, and is heavily inspired by it. Postprocessing filters can be defined in component repositories alongside the implementation of the Kapitan component. This proposal defines Jsonnet as the only templating language which is available for postprocessing filters. Postprocessing filters have access to the Kapitan inventory in the same way as the Kapitan component templates. in addition to inventory access, postprocessing filters are provided with a method to load the contents of a YAML file into Jsonnet. A postprocessing filter should have the same structure as a Kapitan Jsonnet template. The engine renders the Jsonnet, and collects all the keys of the resulting JSON document. The engine then writes the content of each key into a file with the same name as the key. The component’s filters.yml can define an output directory per filter which is used as the base directory for the file names returned by the filter.

Kapitan’s secrets management is leveraged to ensure no secrets appear in any Git repository (input or output) and the in-cluster GitOps engine is configured to run kapitan refs --reveal on the cluster catalog before applying the manifests to the cluster. The Kapitan secrets management is configured to retrieve the secrets from Vault. The Vault instance is configurable in the inventory.

The inventory is collected from various configuration Git repositories, which are either global or in the scope of a customer. A customer may have one or more configuration repositories for their different SYN platforms. The inventory repositories aren’t versioned, as they always reflect the desired state. It must be possible to pull global and customer-specific repositories from different Git hosting platforms.

Inventory repositories are cloned directly into the Kapitan inventory directory to make their contents available as inventory classes in Kapitan.

Component versions are tracked in the inventory parameters in key component_versions and default to master. An example:

Component versions can be any git tree-ish.

Commodore automates the more tedious parts of Kapitan’s secret management, allowing users to simply specify Kapitan secret references (denoted by ?{…​}) in the configuration parameters. Commodore currently only supports Vault KV as back-end for storing secrets. Commodore eliminates the need for users to manually create Kapitan secret files (using kapitan refs --write …​), by scanning the configuration parameters (everything defined under parameters: in Kapitan classes) for secret references and generating secret files in catalog/refs before running kapitan compile. This ensures that the secret files and the catalog are always in sync. All secret references MUST be made in the configuration parameters, otherwise Commodore can’t discover them. Secret references can use reclass references to define dynamic defaults, for example ?{vaultkv:${cluster:tenant}/${cluster:name}/thesecret/thekey}.

Currently the engine is implemented in Python. The implementation provides a local mode where it reuses the existing (probably downloaded by the engine) directory structure to run only local operations. This allows local development of new components without having to re-download all the information on every run of the engine.

The engine uses the Python Jsonnet library to render the postprocessing filters and provides native callbacks to Jsonnet for accessing the Kapitan inventory and for loading YAML files in Jsonnet.

Component versions are extracted from the Kapitan inventory using Kapitan’s inventory_reclass method.