Commodore’s operation can be separated into three rough stages: dependency fetching, catalog compilation, and secrets management.

Apart from these stages of operation, we also document other architectural choices in this page.

Dependency fetching

Currently, the first part of Commodore’s operation revolves around fetching all the dependencies required to compile a catalog. Dependency fetching is implemented using Git repositories and git clone and calls to the Lieutenant API.

Commodore fetches the following dependencies:

  • Static and dynamic cluster facts from Lieutenant API

  • Global configuration

  • Tenant configuration

  • Configuration packages as discovered in global and tenant configuration

  • Components as discovered in global configuration, tenant configuration, and configuration packages

  • Jsonnet libraries as described in the configuration hierarchy

Dependency discovery and versions

To discover all required dependencies (configuration packages and components), Commodore reads the applications array which is made available by reclass. If a dependency should be disabled in a subset of the hierarchy, it can be removed from the applications array by adding the dependency name prefixed with a ~.

  - ~dependency-to-remove

Note that this only works to remove dependencies which have been included previously, and won’t remove dependencies that are included further down in the hierarchy.

Commodore uses the prefix pkg. to distinguish configuration package dependencies from component dependencies.

Commodore currently has no mechanism to automatically discover dependencies based on their names. Instead all components which are referenced in the applications array must be listed in key parameters.components in the hierarchy. Analogously, all configuration packages which are referenced in the applications array must be listed in key parameters.packages in the hierarchy.

Using the configuration hierarchy for specifying component and configuration package locations and versions allows tight integration of dependency management with the rest of the configuration.

The keys parameters.components and parameters.packages each hold a dictionary of dictionaries mapping dependency names to their remote repository location and version. Commodore supports dependencies stored in sub-paths of repositories. The remote repository location is specified in key url. The version is specified in key version. The version can be any Git tree-ish. Commodore will exit with an error if no version is given for a dependency. The path in the repository is specified in key path. If key path isn’t provided, Commodore assumes that the dependency is stored in the repository root.

Commodore uses Git worktrees to make dependencies available in dependencies/<dependency-name>. This allows Commodore to ensure that each remote repository is cloned exactly once. The initial clone of repository is created in dependencies/.repos/ as a bare checkout. For each dependency which is stored in this repository, Commodore ensures that the Git worktree in dependencies/<dependency-name> exists and is checked out with the specified version. Regardless of the value of key path, Commodore creates a checkout of the complete repository in dependencies/<dependency-name>. However, Commodore will create a symlink to the specified path when making the dependency available in the hierarchy.

Commodore will make all included packages available before discovering and fetching components. This ensures that configuration packages can include additional components and can customize component versions.

Commodore will read parameters.components from the hierarchy before component defaults are included.

Example dependency definitions
      version: v4.3.1
      version: v1.0.2

      version: main (1)
      path: package (2)
1 Versions can be any Git tree-ish. This example uses the repository’s main branch as the version
2 Dependencies can be stored in sub-paths of the repository

Commodore will attempt to transform HTTP(S) Git URLs to their SSH-based counterparts when configuring the push URL on the local repository. This transformation allows authorized users to push feature branches to component repos without having to first manually adjust the repository’s push URL. User information (user@ or user:pass@) and non-standard ports in HTTP(S) URLs will be removed when transforming the URL to a SSH-based push URL.

The transformation assumes that SSH URLs follow the pattern git@host:path/to/repo.git (or ssh://git@host/path/to/repo.git). This assumption holds for many popular Git hosting services, such as GitHub and GitLab.

Component and configuration package repositories and versions can be overridden by setting the keys url and versions respectively in parameters.components.<component-name> or parameters.packages.<pacakge-name> in the inventory repositories. This allows configuring a subset of managed clusters to use a fork or different version of a component.

      version: v1.0.0

Since Commodore won’t re-read parameters.components or parameters.packages after including the discovered components' default classes, entries in parameters.components or parameters.packages in a component’s defaults.yml will be ignored.

Commodore will raise an error if it finds entries in parameters.components or parameters.packages which only have the version key. This error is intended to protect users from typos in component names when configuring version overrides.

Component instantiation

With SDD0025, we’ve introduced support for instantiating components multiple times. As discussed in the design document, component authors must explicitly declare that their component supports instantiation. Components advertise that they support instantiation by setting the field multi_instance in parameters.<component_name>._metadata to true. Commodore will exit with an error if a hierarchy tries to instantiate a component which doesn’t explicitly advertise multi-instance support.

Components which are generated with commodore component new by Commodore v0.7.0 or newer already have field _metadata in their component parameters. The component template prefixes the field with an equals sign, which makes the field constant. This ensures that the hierarchy can’t change the contents of the field. See the Kapitan reclass documentation for more details on constant parameters.

Component instance names aren’t namespaced per component, but must be globally unique. Commodore will exit with an error if the hierarchy uses the same instance name twice.

Component instances are declared in the applications array using as as the instantiation keyword. The current implementation of instances can be seen as a mechanism for introducing aliases for a component. Commodore supports hierarchies which include the same component non-aliased and aliased.

Non-aliased components are internally transformed into the aliased identity form component as component. This enables support for hierarchies which want to include a component only using aliases.

A component can be aliased to its own name, regardless of whether the component supports instantiation. Having a component explicitly included both as component and component as component will result in an error during compilation.

The merged content of parameters.<component_name> in the configuration hierarchy is used as the base configuration for each instance. If an instance-aware component is included non-aliased, that "instance" sees the merged content of parameters.<component_name> in the hierarchy. For all other instances of a component, the content of parameters.<instance_name> is merged into parameters.<component_name>. Commodore always sets the meta-parameter parameters._instance to the instance name. For non-aliased instances of instance-aware components, parameters._instance is set to the component name.

Let’s take the configuration below, which includes component nfs-subdir-external-provisioner twice, once non-aliased, and once aliased to nfs-2, as an example. In this example, we’ll end up with two instances of nfs-subdir-external-provisioner, which create volumes on and respectively.

Commodore will apply the usual rules for the relationship between alias name and alias parameters key. Therefore the parameters key for an aliased component is the alias name, but with all dashes replaced by underscores.

  - nfs-subdir-external-provisioner
  - nfs-subdir-external-provisioner as nfs-2
        path: /path/to/share-1
        path: /path/to/share-2

Similar to Helm charts, the components themselves must make sure to not cause any naming collisions of objects belonging to different instances. This is required both for namespaced and non-namespaced resources. Components can make use of the meta-parameter _instance to ensure objects don’t collide, as that parameter is guaranteed to be unique to each instance.

Component dependencies

Components can specify their dependencies in a jsonnetfile.json. Commodore uses jsonnet-bundler to fetch component dependencies.

Components can optionally specify their dependencies in a jsonnetfile.jsonnet. In this case, Commodore renders the jsonnetfile.jsonnet into jsonnetfile.json before running jsonnet-bundler.

Commodore injects the key parameters.<component_name>.jsonnetfile_parameters as external variables when rendering the jsonnetfile.jsonnet.

Jsonnet external variables must be string-valued. Therefore it’s not possible to simply pass the full parameters.component_name as external variables.

Below a jsonnetfile.jsonnet and corresponding class/defaults.yml for component rancher-monitoring are shown. The rancher-monitoring component depends on the kube-prometheus Jsonnet library, but requires different versions of the library depending on the target cluster’s Kubernetes version.

  version: 1,
  dependencies: [
      source: {
        git: {
          remote: '',
          subdir: 'jsonnet/kube-prometheus',
      version: std.extVar('kube_prometheus_version'),
  legacyImports: true,
      '1.17': 4e7440f742df31cd6da188f52ddc4e4037b81599
      '1.18': f69ff3d63de17f3f52b955c3b7e0d7aff0372873
      # Default to K8s 1.18 if not overridden by cluster version
      kube_prometheus_version: ${rancher_monitoring:kube_prometheus_version:1.18}

Component template libraries

Some documentation may refer to component template libraries as "component libraries."

Components can optionally provide Jsonnet template libraries which can be used by other components. To make template libraries available to other components, they must be placed in directory lib/. Commodore enforces that all component libraries are prefixed with the component name.

Components can advertise library aliases in parameter ._metadata.library_aliases. Commodore expects entries of the form alias.libsonnet: target.libsonnet in this parameter.

This allows components to provide implementations for generic library interfaces. For example, cluster monitoring components for different Kubernetes distributions could provide libraries which implement the same interface. In this example, the interface would define functions which other components can use to ensure their alerts are picked up correctly by the cluster’s monitoring stack.

If multiple components advertise the same component alias or if a component advertises an alias which is prefixed with the name of another component deployed on the cluster (the list of deployed components is extracted from applications), Commodore aborts the compilation with an error.

Commodore allows a component c2 which replaces a deprecated component c1 to take over its predecessor’s library prefix if certain conditions are met: In order to be allowed to use its predecessor’s prefix, component c2 needs to explicitly specify that it replaces c1 by setting _metadata.replaces: c1. Additionally, component c1 must either not be deployed on the same cluster, or must be marked as deprecated via _metadata.deprecated: true and must nominate c2 as their replacement by setting _metadata.replaced_by: c2.

Below, a hypothetical example showing component rancher-monitoring advertising library alias alerts.libsonnet is given.

        alerts.libsonnet: rancher-monitoring-alerts.libsonnet
// Implementation omitted

  NamespaceLabels: { (1)
    SYNMonitoring: 'main'
  FormatAlertRule: formatAlertRule, (2)
  FilterAlertRules: filterAlertRules, (3)
1 The set of labels which must be added to a namespace in order for the rancher-monitoring Prometheus to pick up custom resources in that namespace.
2 A function which formats Prometheus alert rules based on the standard alert format defined by rancher-monitoring.
3 A function which filters Prometheus alert rules based on the configuration of component rancher-monitoring.

In this example, the exported fields of lib/rancher-monitoring-alerts.libsonnet match the fields which the alerts.libsonnet interface expects.

Commodore currently doesn’t provide support for component authors to specify library interfaces explicitly.

It’s the responsibility of component authors to agree on an interface and to ensure that their implementations adhere to the interface.

Catalog Compilation

Commodore uses Kapitan to compile the cluster catalog. Commodore defines a Kapitan target for each component instance. Kapitan is called with a few options enabled. Most importantly, Kapitan is configured to support fetching dependencies of components, such as Helm charts. Further, Kapitan is configured with an extended search path to support component libraries and the builtin commodore.libjsonnet. Finally, Kapitan is also configured to search for secret reference files in catalog/refs during compilation. See section Secrets Management for more details on the secrets management implemented with Commodore and Kapitan.

Postprocessing filters

After running Kapitan, Commodore applies postprocessing filters to the output of Kapitan. Postprocessing filters allow components to describe transformations that should be applied to the rendered manifests of the component. Commodore supports two types of postprocessing filters: builtin filters and jsonnet filters. Builtin filters are defined by Commodore itself. Commodore currently provides a single builtin filter helm_namespace which is intended to be used on files generated by the Kapitan helm plugin.

Postprocessing filters are defined in the component class in key parameters.commodore.postprocess.filters. This key is expected to hold a list of filter definitions. Each filter definition is an object, which must have keys type, path and filter.

The field type defines whether the filter definition refers to a builtin or jsonnet filter. The field path indicates the directory on which the filter operates. The field filter defines which filter to apply.

For builtin filters, the filter field holds the name of the builtin filter.

For jsonnet filters, the filter field holds a the path to the jsonnet file defining the filter. The path to the jsonnet filter is relative to the component repository.

The field path is interpreted relative to the component instance’s Kapitan output, which is always in compiled/<instance-name>. Therefore, field path needs to use the same prefix as is used for the entry in parameters.kapitan.compile for which the postprocessing filter should be applied.

Filters can be disabled by setting the optional field enabled in the filter definition to false. If this field isn’t present, filters are treated as enabled.

A component can use the helm_namespace filter by providing the following filter configuration:

  kapitan: ...
        - path: metrics-server/01_helmchart/metrics-server/templates
          type: builtin
          filter: helm_namespace
            namespace: ${metrics_server:namespace}
            create_namespace: true

Secrets Management

Commodore makes use of Kapitan’s secrets management capabilities, but currently only supports references to secrets in Vault (called "Vaultkv" in the Kapitan documentation).

Commodore takes care of generating secret reference files for any secret references (denoted by ?{vaultkv:…​}) found in key parameters in all the classes included by the Kapitan cluster target. Secret references can use reclass references to define dynamic defaults, as Commodore searches for secret references in the rendered reclass inventory.

Commodore saves the generated reference files are stored in the cluster catalog in directory refs/. This directory is configured as the base path in which Kapitan searches for reference files during compilation, allowing references in the inventory to omit the catalog/refs prefix which the would have to include otherwise.

Because Commodore manages the secret files, it can guarantee that the secret files and the catalog are always in sync. All secret references MUST be made in the configuration parameters, otherwise Commodore can’t discover them. Additionally, compiled manifests MUST include the secret reference in clear text, for example by setting stringData for secret objects, as the secret revealing mechanism can’t find the references if they’re already base64 encoded.

Secret file generation

Commodore generates the secret files and their contents according to specific rules. A Kapitan secret reference, for example ?{vaultkv:path/to/secret/thekey}, always refers to a key named thekey in a secret named path/to/secret in Vault’s KV back-end. The address of the Vault instance and the name of the back-end are configurable:

    # Name of the back-end (called mount in Vault)
    vault_mount: kv

For the secret reference mentioned above, Commodore generates a Kapitan secret file in catalog/refs/path/to/secret/thekey with path/to/secret:thekey as the reference to the Vault secret.

Kapitan’s vaultkv secret engine is configured in the class global.common under the dict secret_management. The configuration defaults to and a back-end with name clusters/kv. This can be overridden at any level of the inventory.