Commodore’s operation can be separated into three rough stages: dependency fetching, catalog compilation, and secrets management.

Apart from these stages of operation, we also document other architectural choices in this page.

Dependency fetching

Currently, the first part of Commodore’s operation revolves around fetching all the dependencies required to compile a catalog. Dependency fetching is implemented using Git repositories and git clone and calls to the Lieutenant API.

Commodore fetches the following dependencies:

  • Cluster facts from Lieutenant API

  • Global configuration

  • Tenant configuration

  • Components as discovered in global and tenant configuration

  • Jsonnet libraries as described in the configuration hierarchy

Component discovery and versions

To discover all required components, Commodore reads the applications array which is made available by reclass. If a component should be disabled in a subset of the hierarchy, it can be removed from the applications array by adding the component name prefixed with a ~.

  - ~component-to-remove

Note that this only works to remove components which have been included previously, and won’t remove components that are included further down in the hierarchy.

Commodore currently has no mechanism to automatically discover components based on their names. Instead all components which are referenced in the applications array must be listed in key parameters.components in the hierarchy. Using the configuration hierarchy for specifying component locations and versions allows tight integration of component management with the rest of the configuration.

Commodore will read parameters.components from the hierarchy before component defaults are included. The key parameters.components holds a dictionary of dictionaries mapping component names to their remote repository location and version. The remote repository location is specified in key url. The version is specified in key version. The version can be any Git tree-ish. If no version is given for a component, Commodore defaults to the remote repository’s default branch for the version. Commodore fetches the remote repository and directly checks out the specified version.

      version: v1.0.2

Commodore will attempt to transform HTTP(S) Git URLs to their SSH-based counterparts when configuring the push URL on the local repository. This transformation allows authorized users to push feature branches to component repos without having to first manually adjust the repository’s push URL. User information (user@ or user:pass@) and non-standard ports in HTTP(S) URLs will be removed when transforming the URL to a SSH-based push URL.

The transformation assumes that SSH URLs follow the pattern git@host:path/to/repo.git (or ssh://git@host/path/to/repo.git). This assumption holds for many popular Git hosting services, such as GitHub and GitLab.

Component repositories and versions can be overridden by setting the keys url and versions respectively in parameters.components.<component-name> in the inventory repositories. This allows configuring a subset of managed clusters to use a fork or different version of a component.

      version: v1.0.0

Since Commodore won’t re-read parameters.components after including the discovered components' default classes, entries in parameters.components in a component’s defaults.yml will be ignored.

Component instantiation

With SDD0025, we’ve introduced support for instantiating components multiple times. As discussed in the design document, component authors must explicitly declare that their component supports instantiation. Components declare that they support instantiation by setting the field multi_instance in parameters.<component_name> to true. Commodore will exit with an error if a hierarchy tries to instantiate a component which hasn’t declared that it supports instantiation.

Component instance names aren’t namespaced per component, but must be globally unique. Commodore will exit with an error if the hierarchy uses the same instance name twice.

Component instances are declared in the applications array using as as the instantiation keyword. The current implementation of instances can be seen as a mechanism for introducing aliases for a component. Commodore supports hierarchies which include the same component non-aliased and aliased.

Non-aliased components are internally transformed into the aliased identity form component as component. This enables support for hierarchies which want to include a component only using aliases.

A component can be aliased to its own name, regardless of whether the component supports instantiation. Having a component explicitly included both as component and component as component will result in an error during compilation.

The merged content of parameters.<component_name> in the configuration hierarchy is used as the base configuration for each instance. If an instance-aware component is included non-aliased, that "instance" sees the merged content of parameters.<component_name> in the hierarchy. For all other instances of a component, the content of parameters.<instance_name> is merged into parameters.<component_name>. Commodore always sets the meta-parameter parameters._instance to the instance name. For non-aliased instances of instance-aware components, parameters._instance is set to the component name.

Let’s take the configuration below, which includes component nfs-subdir-external-provisioner twice, once non-aliased, and once aliased to nfs-2, as an example. In this example, we’ll end up with two instances of nfs-subdir-external-provisioner, which create volumes on and respectively.

Commodore will apply the usual rules for the relationship between alias name and alias parameters key. Therefore the parameters key for an aliased component is the alias name, but with all dashes replaced by underscores.

  - nfs-subdir-external-provisioner
  - nfs-subdir-external-provisioner as nfs-2
        path: /path/to/share-1
        path: /path/to/share-2

Similar to Helm charts, the components themselves must make sure to not cause any naming collisions of objects belonging to different instances. This is required both for namespaced and non-namespaced resources. Components can make use of the meta-parameter _instance to ensure objects don’t collide, as that parameter is guaranteed to be unique to each instance.

Component dependencies

Components can specify their dependencies in a jsonnetfile.json. Commodore uses jsonnet-bundler to fetch component dependencies.

Components can optionally specify their dependencies in a jsonnetfile.jsonnet. In this case, Commodore renders the jsonnetfile.jsonnet into jsonnetfile.json before running jsonnet-bundler.

Commodore injects the key parameters.<component_name>.jsonnetfile_parameters as external variables when rendering the jsonnetfile.jsonnet.

Jsonnet external variables must be string-valued. Therefore it’s not possible to simply pass the full parameters.component_name as external variables.

Below a jsonnetfile.jsonnet and corresponding class/defaults.yml for component rancher-monitoring are shown. The rancher-monitoring component depends on the kube-prometheus Jsonnet library, but requires different versions of the library depending on the target cluster’s Kubernetes version.

  version: 1,
  dependencies: [
      source: {
        git: {
          remote: '',
          subdir: 'jsonnet/kube-prometheus',
      version: std.extVar('kube_prometheus_version'),
  legacyImports: true,
      '1.17': 4e7440f742df31cd6da188f52ddc4e4037b81599
      '1.18': f69ff3d63de17f3f52b955c3b7e0d7aff0372873
      # Default to K8s 1.18 if not overridden by cluster version
      kube_prometheus_version: ${rancher_monitoring:kube_prometheus_version:1.18}

Catalog Compilation

Commodore uses Kapitan to compile the cluster catalog. Commodore defines a Kapitan target for each component instance. Kapitan is called with a few options enabled. Most importantly, Kapitan is configured to support fetching dependencies of components, such as Helm charts. Further, Kapitan is configured with an extended search path to support component libraries and the builtin commodore.libjsonnet. Finally, Kapitan is also configured to search for secret reference files in catalog/refs during compilation. See section Secrets Management for more details on the secrets management implemented with Commodore and Kapitan.

Postprocessing filters

After running Kapitan, Commodore applies postprocessing filters to the output of Kapitan. Postprocessing filters allow components to describe transformations that should be applied to the rendered manifests of the component. Commodore supports two types of postprocessing filters: builtin filters and jsonnet filters. Builtin filters are defined by Commodore itself. Commodore currently provides a single builtin filter helm_namespace which is intended to be used on files generated by the Kapitan helm plugin.

Postprocessing filters are defined in the component class in key parameters.commodore.postprocess.filters. This key is expected to hold a list of filter definitions. Each filter definition is an object, which must have keys type, path and filter.

The field type defines whether the filter definition refers to a builtin or jsonnet filter. The field path indicates the directory on which the filter operates. The field filter defines which filter to apply.

For builtin filters, the filter field holds the name of the builtin filter.

For jsonnet filters, the filter field holds a the path to the jsonnet file defining the filter. The path to the jsonnet filter is relative to the component repository.

Filters can be disabled by setting the optional field enabled in the filter definition to false. If this field isn’t present, filters are treated as enabled.

A component can use the helm_namespace filter by providing the following filter configuration:

  kapitan: ...
        - path: metrics-server/01_helmchart/metrics-server/templates
          type: builtin
          filter: helm_namespace
            namespace: ${metrics_server:namespace}
            create_namespace: true

Secrets Management

Commodore makes use of Kapitan’s secrets management capabilities, but currently only supports references to secrets in Vault (called "Vaultkv" in the Kapitan documentation).

Commodore takes care of generating secret reference files for any secret references (denoted by ?{vaultkv:…​}) found in key parameters in all the classes included by the Kapitan cluster target. Secret references can use reclass references to define dynamic defaults, as Commodore searches for secret references in the rendered reclass inventory.

Commodore saves the generated reference files are stored in the cluster catalog in directory refs/. This directory is configured as the base path in which Kapitan searches for reference files during compilation, allowing references in the inventory to omit the catalog/refs prefix which the would have to include otherwise.

Because Commodore manages the secret files, it can guarantee that the secret files and the catalog are always in sync. All secret references MUST be made in the configuration parameters, otherwise Commodore can’t discover them. Additionally, compiled manifests MUST include the secret reference in clear text, for example by setting stringData for secret objects, as the secret revealing mechanism can’t find the references if they’re already base64 encoded.

Secret file generation

Commodore generates the secret files and their contents according to specific rules. A Kapitan secret reference, for example ?{vaultkv:path/to/secret/thekey}, always refers to a key named thekey in a secret named path/to/secret in Vault’s KV back-end. The address of the Vault instance and the name of the back-end are configurable:

    # Name of the back-end (called mount in Vault)
    vault_mount: kv

For the secret reference mentioned above, Commodore generates a Kapitan secret file in catalog/refs/path/to/secret/thekey with path/to/secret:thekey as the reference to the Vault secret.

Kapitan’s vaultkv secret engine is configured in the class global.common under the dict secret_management. The configuration defaults to and a back-end with name clusters/kv. This can be overridden at any level of the inventory.