Architecture

To discover all required dependencies (configuration packages and components), Commodore reads the applications array which is made available by reclass. If a dependency should be disabled in a subset of the hierarchy, it can be removed from the applications array by adding the dependency name prefixed with a ~.

Note that this only works to remove dependencies which have been included previously, and won’t remove dependencies that are included further down in the hierarchy.

Commodore uses the prefix pkg. to distinguish configuration package dependencies from component dependencies.

Commodore currently has no mechanism to automatically discover dependencies based on their names. Instead all components which are referenced in the applications array must be listed in key parameters.components in the hierarchy. Analogously, all configuration packages which are referenced in the applications array must be listed in key parameters.packages in the hierarchy.

Using the configuration hierarchy for specifying component and configuration package locations and versions allows tight integration of dependency management with the rest of the configuration.

The keys parameters.components and parameters.packages each hold a dictionary of dictionaries mapping dependency names to their remote repository location and version. Commodore supports dependencies stored in sub-paths of repositories. The remote repository location is specified in key url. The version is specified in key version. The version can be any Git tree-ish. Commodore will exit with an error if no version is given for a dependency. The path in the repository is specified in key path. If key path isn’t provided, Commodore assumes that the dependency is stored in the repository root.

Commodore uses Git worktrees to make dependencies available in dependencies/<dependency-name>. This allows Commodore to ensure that each remote repository is cloned exactly once. The initial clone of repository git.example.com/path/to/repo.git is created in dependencies/.repos/git.example.com/path/to/repo.git as a bare checkout. For each dependency which is stored in this repository, Commodore ensures that the Git worktree in dependencies/<dependency-name> exists and is checked out with the specified version. Regardless of the value of key path, Commodore creates a checkout of the complete repository in dependencies/<dependency-name>. However, Commodore will create a symlink to the specified path when making the dependency available in the hierarchy.

Commodore will make all included packages available before discovering and fetching components. This ensures that configuration packages can include additional components and can customize component versions.

Commodore will read parameters.components from the hierarchy before component defaults are included.

Component and configuration package repositories and versions can be overridden by setting the keys url and versions respectively in parameters.components.<component-name> or parameters.packages.<pacakge-name> in the inventory repositories. This allows configuring a subset of managed clusters to use a fork or different version of a component.

With SDD0025, we’ve introduced support for instantiating components multiple times. As discussed in the design document, component authors must explicitly declare that their component supports instantiation. Components advertise that they support instantiation by setting the field multi_instance in parameters.<component_name>._metadata to true. Commodore will exit with an error if a hierarchy tries to instantiate a component which doesn’t explicitly advertise multi-instance support.

Component instance names aren’t namespaced per component, but must be globally unique. Commodore will exit with an error if the hierarchy uses the same instance name twice.

Component instances are declared in the applications array using as as the instantiation keyword. The current implementation of instances can be seen as a mechanism for introducing aliases for a component. Commodore supports hierarchies which include the same component non-aliased and aliased.

The merged content of parameters.<component_name> in the configuration hierarchy is used as the base configuration for each instance. If an instance-aware component is included non-aliased, that "instance" sees the merged content of parameters.<component_name> in the hierarchy. For all other instances of a component, the content of parameters.<instance_name> is merged into parameters.<component_name>. Commodore always sets the meta-parameter parameters._instance to the instance name. For non-aliased instances of instance-aware components, parameters._instance is set to the component name.

Let’s take the configuration below, which includes component nfs-subdir-external-provisioner twice, once non-aliased, and once aliased to nfs-2, as an example. In this example, we’ll end up with two instances of nfs-subdir-external-provisioner, which create volumes on nfs.example.org:/path/to/share-1 and nfs.example.org:/path/to/share-2 respectively.

Similar to Helm charts, the components themselves must make sure to not cause any naming collisions of objects belonging to different instances. This is required both for namespaced and non-namespaced resources. Components can make use of the meta-parameter _instance to ensure objects don’t collide, as that parameter is guaranteed to be unique to each instance.

Components can specify their dependencies in a jsonnetfile.json. Commodore uses jsonnet-bundler to fetch component dependencies.

Components can optionally specify their dependencies in a jsonnetfile.jsonnet. In this case, Commodore renders the jsonnetfile.jsonnet into jsonnetfile.json before running jsonnet-bundler.

Commodore injects the key parameters.<component_name>.jsonnetfile_parameters as external variables when rendering the jsonnetfile.jsonnet.

Below a jsonnetfile.jsonnet and corresponding class/defaults.yml for component rancher-monitoring are shown. The rancher-monitoring component depends on the kube-prometheus Jsonnet library, but requires different versions of the library depending on the target cluster’s Kubernetes version.

Components can optionally provide Jsonnet template libraries which can be used by other components. To make template libraries available to other components, they must be placed in directory lib/. Commodore enforces that all component libraries are prefixed with the component name.

Components can advertise library aliases in parameter ._metadata.library_aliases. Commodore expects entries of the form alias.libsonnet: target.libsonnet in this parameter.

This allows components to provide implementations for generic library interfaces. For example, cluster monitoring components for different Kubernetes distributions could provide libraries which implement the same interface. In this example, the interface would define functions which other components can use to ensure their alerts are picked up correctly by the cluster’s monitoring stack.

If multiple components advertise the same component alias or if a component advertises an alias which is prefixed with the name of another component deployed on the cluster (the list of deployed components is extracted from applications), Commodore aborts the compilation with an error.

Commodore allows a component c2 which replaces a deprecated component c1 to take over its predecessor’s library prefix if certain conditions are met: In order to be allowed to use its predecessor’s prefix, component c2 needs to explicitly specify that it replaces c1 by setting _metadata.replaces: c1. Additionally, component c1 must either not be deployed on the same cluster, or must be marked as deprecated via _metadata.deprecated: true and must nominate c2 as their replacement by setting _metadata.replaced_by: c2.

Below, a hypothetical example showing component rancher-monitoring advertising library alias alerts.libsonnet is given.

In this example, the exported fields of lib/rancher-monitoring-alerts.libsonnet match the fields which the alerts.libsonnet interface expects.

After running Kapitan, Commodore applies postprocessing filters to the output of Kapitan. Postprocessing filters allow components to describe transformations that should be applied to the rendered manifests of the component. Commodore supports two types of postprocessing filters: builtin filters and jsonnet filters. Builtin filters are defined by Commodore itself. Commodore currently provides a single builtin filter helm_namespace which is intended to be used on files generated by the Kapitan helm plugin.

Postprocessing filters are defined in the component class in key parameters.commodore.postprocess.filters. This key is expected to hold a list of filter definitions. Each filter definition is an object, which must have keys type, path and filter.

The field type defines whether the filter definition refers to a builtin or jsonnet filter. The field path indicates the directory on which the filter operates. The field filter defines which filter to apply.

For builtin filters, the filter field holds the name of the builtin filter.

For jsonnet filters, the filter field holds a the path to the jsonnet file defining the filter. The path to the jsonnet filter is relative to the component repository.

The field path is interpreted relative to the component instance’s Kapitan output, which is always in compiled/<instance-name>. Therefore, field path needs to use the same prefix as is used for the entry in parameters.kapitan.compile for which the postprocessing filter should be applied.

Filters can be disabled by setting the optional field enabled in the filter definition to false. If this field isn’t present, filters are treated as enabled.

A component can use the helm_namespace filter by providing the following filter configuration:

Commodore generates the secret files and their contents according to specific rules. A Kapitan secret reference, for example ?{vaultkv:path/to/secret/thekey}, always refers to a key named thekey in a secret named path/to/secret in Vault’s KV back-end. The address of the Vault instance and the name of the back-end are configurable:

For the secret reference mentioned above, Commodore generates a Kapitan secret file in catalog/refs/path/to/secret/thekey with path/to/secret:thekey as the reference to the Vault secret.

Kapitan’s vaultkv secret engine is configured in the class global.common under the dict secret_management. The configuration defaults to vault-prod.syn.vshn.net and a back-end with name clusters/kv. This can be overridden at any level of the inventory.