SDD 0021 - Cluster Catalog Compilation

In order for Quartermaster to figure out which clusters are affected by a certain config change, the Kapitan inventory needs to be parsed. Kapitan does this by using reclass which is a tool for merging and searching data sources recursively in YAML files.

Quartermaster can use reclass to parse the inventory and search for clusters that are affected by changes in a certain class. One issue we’ve to work around in this approach is, that reclass can only parse the inventory if it’s complete and all referenced classes are available. This is a problem since creating the whole inventory (with all components) would be too resource and time intensive. As a workaround Quartermaster should only have to get the global and tenant parts of the inventory. The classes for the different Commodore components should be faked and generated as empty files. The only downside to this approach is that if a component class imports another class, this won’t be captured by Quartermaster which might result in cluster catalogs not being compiled. Since we currently don’t allow components to include classes this isn’t a problem.

Triggering a webhook can be implemented in either a Git repo natively (for example GitHub Webhooks) or via a CI/CD system (for example a GitLab CI job which runs curl). A webhook can also be called by any other tool or even manually with a curl command.

A webhook can include the following optional information:

To prevent exsessive resource usage or even DoS attacks, the webhooks need to be authenticated. A permission system needs to be in place to decide which entities are allowed to trigger which actions. It shouldn’t be possible for example for a tenant to trigger compilation for clusters of other tenants.

To properly monitor the compilation jobs, metrics must be generated and stored for each job. This can be done by using the Kubernetes metrics already available for the status of each job. Additionally, each compilation job pushes detailed metrics about the status to a Prometheus Pushgateway. This allows us to create detailed monitoring and alerting rules, even for jobs running on different clusters.

The compilation engine is also responsible for creating an SSH keypair to be used by the compile job. The public key of which needs to be configured with read access on the global and tenant repositories and with write access on the cluster catalog repository.

If configuration in the global inventory repo changes, Quartermaster needs to find all clusters which are affected by the changed configuration.

If configuration in a tenant inventory repo changes, Quartermaster needs to find all clusters which are affected by the changed configuration. This will be the same mechanism as for config changes in the global config repo. The set of clusters to consider are the clusters of the respective tenant.

If the facts of a cluster change the set of clusters to consider already only consists of this cluster. Triggering the webhook needs to be implemented in Lieutenant (operator or API).

The new release of a Commodore component won’t trigger anything directly. If the specific version for a component is being updated it results in a config change in either the global inventory repo or a tenant repo. This in turn results in the recompilation of the affected clusters.

To manually trigger a compile for developping or debugging purposes the webhook can be called with a tool like curl. Providing a cluster or tenant ID can limit the set of clusters to compile.

Since the compilation is a relatively expensive operation resource wise, it might become necessary to implement some kind of rate limiting. For example Quartermaster could make sure that a certain amount of compile jobs per amount of time isn’t exceeded for a single cluster or tenant. This would mitigate the situation where a lot of changes in various Git repos would result in many compile jobs which could exhaust the resources of a cluster. The downside would be a longer reaction time for changes to be applied on a cluster.