ArgoCD multi-tenancy

Project Syn allows users to assign Commodore components installed on a cluster to multiple teams. The original design of this feature is documented in SDD 0030 - Project Syn ArgoCD multi-tenant support.

Configuration

The assignment of components to teams is controlled via inventory parameter syn. In particular, parameters syn.owner and syn.teams describe which team is responsible for which component.

Each component instance on a cluster must be assigned to exactly one team. To explicitly assign an instance to a team, the instance name is added to syn.teams.<team>.instances. Instances can be removed from syn.teams.<team>.instances by adding the instance name prefixed with ~ ^[1] on the list.

The syn-teams.libsonnet Jsonnet library exposes fields which provide rendered views of the team to instance and instance to team mappings.

Any instance that’s not explicitly assigned to a team is assigned to the cluster’s owner team which is indicated by parameter syn.owner.

To assign instance-aware components which deploy all instances through a single ArgoCD application to a non-owner team, the component name must be added to syn.teams.<team>.instances in order to assign the application to that team.

Implementation

Project Syn implements ArgoCD multi-tenancy by deploying an independent ArgoCD project and root application ^[2] for each team that’s responsible for at least one component present on the cluster.

Instances assigned to the cluster’s owner team remain assigned to ArgoCD project syn and root application root. This project and root application are always present on a Project Syn-managed cluster regardless of whether syn.owner is set.

For each additional team that’s responsible for one or more component instances on the cluster, a project called <team-name> and a root application called root-<team-name> are present on the cluster.

To ensure that each project and root application are fully independent, each root application reads ArgoCD application manifests from a different path in the cluster catalog. The default root application reads manifests from catalog path apps/. All other root applications read manifests from catalog path apps-<team-name>/.

Additional projects and root applications are bootstrapped by Steward based on the list of team names present in a config map (by default additional-root-apps). Steward reads this config map once a minute and ensures that an ArgoCD project and root app exist for each team.

The ArgoCD component dynamically sets spec.project of ArgoCD application manifests by treating .metadata.name of the manifest as a component instance name and looking up the responsible team for that instance through the syn-teams.libsonnet Jsonnet library.

The ArgoCD component library assumes that each ArgoCD application’s metadata.name is set to either a component name or a component instance name.

To ease the transition to the multi-tenancy model, Commodore components need to be explicitly marked as multi-tenant aware by setting the component parameter _metadata.multi_tenant to true. The ArgoCD commodore component will raise an error if a component which isn’t marked as multi-tenant-aware is assigned to a team other than the cluster’s owner team.

Components that set _metadata.multi_tenant=true are responsible for making sure that they write their ArgoCD application manifests into the appropriate catalog path.

The ArgoCD component ensures that spec.project of an application manifest generated by the argocd.libjsonnet component library will be set to the responsible team’s ArgoCD project. As noted above, each non-owner team’s ArgoCD project is called <team-name>. This allows components to make sure that the application manifests are written to the correct catalog path by setting output_path: . for the Kapitan compile step for component/app.jsonnet and by rendering each application’s output key based on the application’s spec.project.

class/component.yml

parameters:
  kapitan:
    compile:
      - input_paths:
          - ${_instance}/component/app.jsonnet
        input_type: jsonnet
        output_path: apps/
        output_path: .

component/app.jsonnet

local kap = import 'lib/kapitan.libjsonnet';
local kube = import 'lib/kube.libjsonnet';
local inv = kap.inventory();
local params = inv.parameters.<component>(1)
local argocd = import 'lib/argocd.libjsonnet';

local app = argocd.App(<component>, params.namespace, secrets=true); (1)

local appPath =
  local project = std.get(app, 'spec', { project: 'syn' }).project; (2)
  if project == 'syn' then 'apps' else 'apps-%s' % project;

{
  ['%s/<component>' % appPath]: app, (1)
}

1	Replace `<component> with the component’s name.
2	We use `std.get()` to read `spec.project` because `commodore component compile` currently uses a fake `argocd.libjsonnet` which doesn’t generate a valid ArgoCD application manifest.

In order to enable users to easily move ownership of existing component instances, Steward and the ArgoCD Commodore component ensure that resource pruning is disabled for each root application.

Without this configuration, there’s a chance that a component instance would be deleted and recreated when it’s transferred to a different team.

Usage in Commodore components

Commodore components that want to use the component team ownership information should use the syn-teams.libsonnet Jsonnet library which provides the ownership information in a variety of ways.

This library is available on GitHub and can be included as a regular component Jsonnet dependency:

jsonnetfile.json

{
  "version": 1,
  "dependencies": [
    {
      "source": {
        "git": {
          "remote": "https://github.com/projectsyn/jsonnet-libs",
          "subdir": ""
        }
      },
      "version": "main",
      "name": "syn"
    }
  ],
  "legacyImports": true
}

Users should check the library for usage documentation.

1. The SDD requires that consumers of the instance list apply com.renderArray()

2. Project Syn uses the term "root application" for ArgoCD’s app of apps concept.