# Architecture

Commodore’s operation can be separated into three rough stages: dependency fetching, catalog compilation, and secrets management.

Apart from these stages of operation, we also document other architectural choices in this page.

## Dependency fetching

Currently, the first part of Commodore’s operation revolves around fetching all the dependencies required to compile a catalog. Dependency fetching is implemented using Git repositories and `git clone` and calls to the Lieutenant API.

Commodore fetches the following dependencies:

• Static and dynamic cluster facts from Lieutenant API

• Global configuration

• Tenant configuration

• Configuration packages as discovered in global and tenant configuration

• Components as discovered in global configuration, tenant configuration, and configuration packages

• Jsonnet libraries as described in the configuration hierarchy

### Dependency discovery and versions

To discover all required dependencies (configuration packages and components), Commodore reads the `applications` array which is made available by reclass. If a dependency should be disabled in a subset of the hierarchy, it can be removed from the `applications` array by adding the dependency name prefixed with a `~`.

``````applications:
- ~dependency-to-remove``````

Note that this only works to remove dependencies which have been included previously, and won’t remove dependencies that are included further down in the hierarchy.

Commodore uses the prefix `pkg.` to distinguish configuration package dependencies from component dependencies.

Commodore currently has no mechanism to automatically discover dependencies based on their names. Instead all components which are referenced in the `applications` array must be listed in key `parameters.components` in the hierarchy. Analogously, all configuration packages which are referenced in the `applications` array must be listed in key `parameters.packages` in the hierarchy.

Using the configuration hierarchy for specifying component and configuration package locations and versions allows tight integration of dependency management with the rest of the configuration.

The keys `parameters.components` and `parameters.packages` each hold a dictionary of dictionaries mapping dependency names to their remote repository location and version. Commodore supports dependencies stored in sub-paths of repositories. The remote repository location is specified in key `url`. The version is specified in key `version`. The version can be any Git tree-ish. Commodore will exit with an error if no version is given for a dependency. The path in the repository is specified in key `path`. If key `path` isn’t provided, Commodore assumes that the dependency is stored in the repository root.

Commodore uses Git worktrees to make dependencies available in `dependencies/<dependency-name>`. This allows Commodore to ensure that each remote repository is cloned exactly once. The initial clone of repository `git.example.com/path/to/repo.git` is created in `dependencies/.repos/git.example.com/path/to/repo.git` as a bare checkout. For each dependency which is stored in this repository, Commodore ensures that the Git worktree in `dependencies/<dependency-name>` exists and is checked out with the specified `version`. Regardless of the value of key `path`, Commodore creates a checkout of the complete repository in `dependencies/<dependency-name>`. However, Commodore will create a symlink to the specified path when making the dependency available in the hierarchy.

Commodore will make all included packages available before discovering and fetching components. This ensures that configuration packages can include additional components and can customize component versions.

Commodore will read `parameters.components` from the hierarchy before component defaults are included.

Example dependency definitions
``````parameters:
components:
argocd:
url: https://github.com/projectsyn/component-argocd.git
version: v4.3.1
metrics-server:
url: https://github.com/projectsyn/component-metrics-server.git
version: v1.0.2

packages:
dummy:
url: https://github.com/projectsyn/dummy.git
version: main (1)
path: package (2)``````
 1 Versions can be any Git tree-ish. This example uses the repository’s `main` branch as the version 2 Dependencies can be stored in sub-paths of the repository
 Commodore will attempt to transform HTTP(S) Git URLs to their SSH-based counterparts when configuring the push URL on the local repository. This transformation allows authorized users to push feature branches to component repos without having to first manually adjust the repository’s push URL. User information (`user@` or `user:pass@`) and non-standard ports in HTTP(S) URLs will be removed when transforming the URL to a SSH-based push URL. The transformation assumes that SSH URLs follow the pattern `git@host:path/to/repo.git` (or `ssh://git@host/path/to/repo.git`). This assumption holds for many popular Git hosting services, such as GitHub and GitLab.

Component and configuration package repositories and versions can be overridden by setting the keys `url` and `versions` respectively in `parameters.components.<component-name>` or `parameters.packages.<pacakge-name>` in the inventory repositories. This allows configuring a subset of managed clusters to use a fork or different version of a component.

``````parameters:
components:
argocd:
url: https://github.com/projectsyn/component-argocd-fork.git
version: v1.0.0``````
 Since Commodore won’t re-read `parameters.components` or `parameters.packages` after including the discovered components' default classes, entries in `parameters.components` or `parameters.packages` in a component’s `defaults.yml` will be ignored.
 Commodore will raise an error if it finds entries in `parameters.components` or `parameters.packages` which only have the `version` key. This error is intended to protect users from typos in component names when configuring version overrides.

### Component instantiation

With SDD0025, we’ve introduced support for instantiating components multiple times. As discussed in the design document, component authors must explicitly declare that their component supports instantiation. Components advertise that they support instantiation by setting the field `multi_instance` in `parameters.<component_name>._metadata` to `true`. Commodore will exit with an error if a hierarchy tries to instantiate a component which doesn’t explicitly advertise multi-instance support.

 Components which are generated with `commodore component new` by Commodore v0.7.0 or newer already have field `_metadata` in their component parameters. The component template prefixes the field with an equals sign, which makes the field constant. This ensures that the hierarchy can’t change the contents of the field. See the Kapitan reclass documentation for more details on constant parameters.

Component instance names aren’t namespaced per component, but must be globally unique. Commodore will exit with an error if the hierarchy uses the same instance name twice.

Component instances are declared in the `applications` array using `as` as the instantiation keyword. The current implementation of instances can be seen as a mechanism for introducing aliases for a component. Commodore supports hierarchies which include the same component non-aliased and aliased.

 Non-aliased components are internally transformed into the aliased identity form `component as component`. This enables support for hierarchies which want to include a component only using aliases. A component can be aliased to its own name, regardless of whether the component supports instantiation. Having a component explicitly included both as `component` and `component as component` will result in an error during compilation.

The merged content of `parameters.<component_name>` in the configuration hierarchy is used as the base configuration for each instance. If an instance-aware component is included non-aliased, that "instance" sees the merged content of `parameters.<component_name>` in the hierarchy. For all other instances of a component, the content of `parameters.<instance_name>` is merged into `parameters.<component_name>`. Commodore always sets the meta-parameter `parameters._instance` to the instance name. For non-aliased instances of instance-aware components, `parameters._instance` is set to the component name.

Let’s take the configuration below, which includes component `nfs-subdir-external-provisioner` twice, once non-aliased, and once aliased to `nfs-2`, as an example. In this example, we’ll end up with two instances of nfs-subdir-external-provisioner, which create volumes on `nfs.example.org:/path/to/share-1` and `nfs.example.org:/path/to/share-2` respectively.

 Commodore will apply the usual rules for the relationship between alias name and alias parameters key. Therefore the parameters key for an aliased component is the alias name, but with all dashes replaced by underscores.
tenant/common.yml
``````applications:
- nfs-subdir-external-provisioner
- nfs-subdir-external-provisioner as nfs-2
parameters:
nfs_subdir_external_provisioner:
helm_values:
nfs:
server: nfs.example.org
path: /path/to/share-1
nfs_2:
helm_values:
nfs:
path: /path/to/share-2``````

Similar to Helm charts, the components themselves must make sure to not cause any naming collisions of objects belonging to different instances. This is required both for namespaced and non-namespaced resources. Components can make use of the meta-parameter `_instance` to ensure objects don’t collide, as that parameter is guaranteed to be unique to each instance.

### Component dependencies

Components can specify their dependencies in a `jsonnetfile.json`. Commodore uses jsonnet-bundler to fetch component dependencies.

Components can optionally specify their dependencies in a `jsonnetfile.jsonnet`. In this case, Commodore renders the `jsonnetfile.jsonnet` into `jsonnetfile.json` before running jsonnet-bundler.

Commodore injects the key `parameters.<component_name>.jsonnetfile_parameters` as external variables when rendering the `jsonnetfile.jsonnet`.

 Jsonnet external variables must be string-valued. Therefore it’s not possible to simply pass the full `parameters.component_name` as external variables.

Below a `jsonnetfile.jsonnet` and corresponding `class/defaults.yml` for component `rancher-monitoring` are shown. The `rancher-monitoring` component depends on the `kube-prometheus` Jsonnet library, but requires different versions of the library depending on the target cluster’s Kubernetes version.

jsonnetfile.jsonnet
``````{
version: 1,
dependencies: [
{
source: {
git: {
remote: 'https://github.com/coreos/kube-prometheus',
subdir: 'jsonnet/kube-prometheus',
},
},
version: std.extVar('kube_prometheus_version'),
},
],
legacyImports: true,
}``````
class/defaults.yml
``````parameters:
rancher_monitoring:
kube_prometheus_version:
'1.17': 4e7440f742df31cd6da188f52ddc4e4037b81599
'1.18': f69ff3d63de17f3f52b955c3b7e0d7aff0372873
jsonnetfile_parameters:
# Default to K8s 1.18 if not overridden by cluster version
kube_prometheus_version: ${rancher_monitoring:kube_prometheus_version:1.18}`````` ### Component template libraries  Some documentation may refer to component template libraries as "component libraries." Components can optionally provide Jsonnet template libraries which can be used by other components. To make template libraries available to other components, they must be placed in directory `lib/`. Commodore enforces that all component libraries are prefixed with the component name. Components can advertise library aliases in parameter `._metadata.library_aliases`. Commodore expects entries of the form `alias.libsonnet: target.libsonnet` in this parameter. This allows components to provide implementations for generic library interfaces. For example, cluster monitoring components for different Kubernetes distributions could provide libraries which implement the same interface. In this example, the interface would define functions which other components can use to ensure their alerts are picked up correctly by the cluster’s monitoring stack. If multiple components advertise the same component alias or if a component advertises an alias which is prefixed with the name of another known component (the list of known components is extracted from `parameters.components`), Commodore aborts the compilation with an error. Below, a hypothetical example showing component `rancher-monitoring` advertising library alias `alerts.libsonnet` is given. class/defaults.yml ``````parameters: rancher_monitoring: =_metadata: library_aliases: alerts.libsonnet: rancher-monitoring-alerts.libsonnet`````` lib/rancher-monitoring-alerts.libsonnet ``````// Implementation omitted { NamespaceLabels: { (1) SYNMonitoring: 'main' }, FormatAlertRule: formatAlertRule, (2) FilterAlertRules: filterAlertRules, (3) }``````  1 The set of labels which must be added to a namespace in order for the `rancher-monitoring` Prometheus to pick up custom resources in that namespace. 2 A function which formats Prometheus alert rules based on the standard alert format defined by `rancher-monitoring`. 3 A function which filters Prometheus alert rules based on the configuration of component `rancher-monitoring`. In this example, the exported fields of `lib/rancher-monitoring-alerts.libsonnet` match the fields which the `alerts.libsonnet` interface expects.  Commodore currently doesn’t provide support for component authors to specify library interfaces explicitly. It’s the responsibility of component authors to agree on an interface and to ensure that their implementations adhere to the interface. ## Catalog Compilation Commodore uses Kapitan to compile the cluster catalog. Commodore defines a Kapitan target for each component instance. Kapitan is called with a few options enabled. Most importantly, Kapitan is configured to support fetching dependencies of components, such as Helm charts. Further, Kapitan is configured with an extended search path to support component libraries and the builtin `commodore.libjsonnet`. Finally, Kapitan is also configured to search for secret reference files in `catalog/refs` during compilation. See section Secrets Management for more details on the secrets management implemented with Commodore and Kapitan. ### Postprocessing filters After running Kapitan, Commodore applies postprocessing filters to the output of Kapitan. Postprocessing filters allow components to describe transformations that should be applied to the rendered manifests of the component. Commodore supports two types of postprocessing filters: builtin filters and jsonnet filters. Builtin filters are defined by Commodore itself. Commodore currently provides a single builtin filter `helm_namespace` which is intended to be used on files generated by the Kapitan helm plugin. Postprocessing filters are defined in the component class in key `parameters.commodore.postprocess.filters`. This key is expected to hold a list of filter definitions. Each filter definition is an object, which must have keys `type`, `path` and `filter`. The field `type` defines whether the filter definition refers to a builtin or jsonnet filter. The field `path` indicates the directory on which the filter operates. The field `filter` defines which filter to apply. For builtin filters, the `filter` field holds the name of the builtin filter. For jsonnet filters, the `filter` field holds a the path to the jsonnet file defining the filter. The path to the jsonnet filter is relative to the component repository. The field `path` is interpreted relative to the component instance’s Kapitan output, which is always in `compiled/<instance-name>`. Therefore, field `path` needs to use the same prefix as is used for the entry in `parameters.kapitan.compile` for which the postprocessing filter should be applied. Filters can be disabled by setting the optional field `enabled` in the filter definition to `false`. If this field isn’t present, filters are treated as enabled. A component can use the `helm_namespace` filter by providing the following filter configuration: component-metrics-server/class/metrics-server.yml ``````parameters: kapitan: ... commodore: postprocess: filters: - path: metrics-server/01_helmchart/metrics-server/templates type: builtin filter: helm_namespace filterargs: namespace:${metrics_server:namespace}
create_namespace: true``````

## Secrets Management

Commodore makes use of Kapitan’s secrets management capabilities, but currently only supports references to secrets in Vault (called "Vaultkv" in the Kapitan documentation).

Commodore takes care of generating secret reference files for any secret references (denoted by `?{vaultkv:…​}`) found in key `parameters` in all the classes included by the Kapitan cluster target. Secret references can use reclass references to define dynamic defaults, as Commodore searches for secret references in the rendered reclass inventory.

Commodore saves the generated reference files are stored in the cluster catalog in directory `refs/`. This directory is configured as the base path in which Kapitan searches for reference files during compilation, allowing references in the inventory to omit the `catalog/refs` prefix which the would have to include otherwise.

Because Commodore manages the secret files, it can guarantee that the secret files and the catalog are always in sync. All secret references MUST be made in the configuration parameters, otherwise Commodore can’t discover them. Additionally, compiled manifests MUST include the secret reference in clear text, for example by setting `stringData` for secret objects, as the secret revealing mechanism can’t find the references if they’re already base64 encoded.

### Secret file generation

Commodore generates the secret files and their contents according to specific rules. A Kapitan secret reference, for example `?{vaultkv:path/to/secret/thekey}`, always refers to a key named `thekey` in a secret named `path/to/secret` in Vault’s KV back-end. The address of the Vault instance and the name of the back-end are configurable:

``````parameters:
secret_management:
For the secret reference mentioned above, Commodore generates a Kapitan secret file in `catalog/refs/path/to/secret/thekey` with `path/to/secret:thekey` as the reference to the Vault secret.
Kapitan’s `vaultkv` secret engine is configured in the class `global.common` under the dict `secret_management`. The configuration defaults to vault-prod.syn.vshn.net and a back-end with name `clusters/kv`. This can be overridden at any level of the inventory.