Getting Started with Project Syn
This guide helps to get started with the various Project Syn tools.
Have a look at features and architecture first to get an idea about Project Syn, this helps to understand what we’re doing here. |
Requirements
Before you start, please make sure to have these requirements available:
-
A local Kubernetes cluster (
k3s
), managed with k3d or Docker Desktop for Mac -
A GitLab.com account with an SSH key configured
-
You can use
ssh-keygen
to generate an SSH key if you don’t yet have one -
Alternatively you could use your own GitLab instance (some adjustments need to be made in the guide)
-
-
The following commands must work on your shell:
curl
,jq
,ssh-keyscan
andbase64
Getting started on Linux with k3d
|
Getting started on macOS with Docker Desktop for Mac
|
Kickstart Lieutenant
Lieutenant is the central inventory service all Project Syn managed clusters report to, it consists of the Lieutenant Operator and the Lieutenant API.
As you can see on the architecture diagram, Lieutenant is the central API which is being used by all other Project Syn tools. Therefore it needs to be installed first so that all other components are actually able to do their job.
Create a namespace to host the Operator and the API
kubectl create namespace lieutenant
Install Lieutenant Operator
Install Lieutenant Operator with the following commands:
# CRDs (global scope)
kubectl apply -k "github.com/projectsyn/lieutenant-operator/deploy/crds?ref=v0.4.2"
# Operator deployment
kubectl -n lieutenant apply -k "github.com/projectsyn/lieutenant-operator/deploy?ref=v0.4.2"
# Operator configuration
kubectl -n lieutenant set env deployment/lieutenant-operator -c lieutenant-operator \
DEFAULT_DELETION_POLICY=Delete \
DEFAULT_GLOBAL_GIT_REPO_URL=https://github.com/projectsyn/getting-started-commodore-defaults \
LIEUTENANT_DELETE_PROTECTION=false \
SKIP_VAULT_SETUP=true
These environment variables will configure Lieutenant Operator to:
-
not set deletion protection annotations (
DEFAULT_DELETION_POLICY
) -
configure the global git repo URL by default on Tenants (
DEFAULT_GLOBAL_GIT_REPO_URL
) -
delete external resources by default (good for cleaning up after this guide is finished) (
LIEUTENANT_DELETE_PROTECTION
) -
not use Hashicorp Vault (don’t do this for production deployments) (
SKIP_VAULT_SETUP
)
Install Lieutenant API
Install Lieutenant API with the following commands:
# API deployment
kubectl -n lieutenant apply -k "github.com/projectsyn/lieutenant-api/deploy?ref=v0.4.0"
# API configuration
kubectl -n lieutenant set env deployment/lieutenant-api -c lieutenant-api \
DEFAULT_API_SECRET_REF_NAME=gitlab-com
# Ingress
if [[ "$OSTYPE" == "darwin"* ]]; then export INGRESS_IP=127.0.0.1; else export INGRESS_IP=$(kubectl -n kube-system get svc traefik -o jsonpath="{.status.loadBalancer.ingress[0].ip}"); fi
kubectl -n lieutenant apply -f -<<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: lieutenant-api
spec:
rules:
- host: lieutenant.${INGRESS_IP}.nip.io
http:
paths:
- path: /
backend:
serviceName: lieutenant-api
servicePort: 80
EOF
Check that the API is accessible:
echo http://lieutenant.${INGRESS_IP}.nip.io/healthz
curl http://lieutenant.${INGRESS_IP}.nip.io/healthz
This should return ok
as the answer to the curl
command. You can see this is the same API Commodore and Steward will use (in this getting started guide without https and using the nip.io dynamic URL).
The API documentation can be accessed in your browser under lieutenant.${INGRESS_IP}.nip.io/docs. |
Prepare Lieutenant Operator access to GitLab
Lieutenant needs API access to a GitLab server. This is required to create and manage Git repositories for clusters and tenants.
What are tenants?
A "tenant" is an entity to assign clusters to. This entity could be a customer, a department, a team, or anything you want to group clusters with. This concept is also used by Commodore, as every tenant gets his own configuration Git repository to (for example) apply common settings to all clusters belonging to a particular tenant. Any cluster specific configuration values are stored in that tenant’s own configuration Git repository. |
Create a Kubernetes secret which contains the access token for the GitLab API, which can be generated here: gitlab.com/profile/personal_access_tokens (needs api
scope, amend gitlab.com with your own GitLab instance URL if needed).
Replace MYTOKEN
with the generated GitLab API token. If you’re using your own GitLab instance, amend GITLAB_ENDPOINT
.
export GITLAB_TOKEN=MYTOKEN
export GITLAB_ENDPOINT=gitlab.com
kubectl -n lieutenant create secret generic gitlab-com \
--from-literal=endpoint="https://${GITLAB_ENDPOINT}" \
--from-literal=hostKeys="$(ssh-keyscan ${GITLAB_ENDPOINT})" \
--from-literal=token=${GITLAB_TOKEN}
Prepare Lieutenant API Authentication and Authorization
As the Lieutenant API uses the underlying Kubernetes cluster for authentication and authorization, the following objects need to be created:
-
Role
-
RoleBinding
-
ServiceAccount
kubectl -n lieutenant apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: lieutenant-api-user
rules:
- apiGroups:
- syn.tools
resources:
- clusters
- clusters/status
- tenants
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: lieutenant-api-user
roleRef:
kind: Role
name: lieutenant-api-user
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: api-access-synkickstart
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: api-access-synkickstart
EOF
Create Lieutenant Objects: Tenant and Cluster
In this section you will create your first Lieutenant configuration objects using the API to test the deployment and configuration.
-
Prepare access to API, replace
MYUSER
with your GitLab usernameexport LIEUTENANT_TOKEN=$(kubectl -n lieutenant get secret $(kubectl -n lieutenant get sa api-access-synkickstart -o go-template='{{(index .secrets 0).name}}') -o go-template='{{.data.token | base64decode}}') export LIEUTENANT_AUTH="Authorization: Bearer ${LIEUTENANT_TOKEN}" export LIEUTENANT_URL="lieutenant.${INGRESS_IP}.nip.io" export GITLAB_USERNAME="MYUSER"
-
Create a Lieutenant Tenant via the API
TENANT_ID=$(curl -s -H "$LIEUTENANT_AUTH" -H "Content-Type: application/json" -X POST --data "{\"displayName\":\"My first Tenant\",\"gitRepo\":{\"url\":\"ssh://git@${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/mytenant\"}}" "http://${LIEUTENANT_URL}/tenants" | jq -r ".id") echo $TENANT_ID echo https://${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/mytenant
If everything went well, the Lieutenant Operator created a new git repository under ${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/mytenant, which will be used to store the configuration used by Commodore to create a catalog for a cluster. -
Patch the Tenant object directly in Kubernetes to add a Cluster template and set the
globalGitRepoURL
.kubectl -n lieutenant patch tenant $TENANT_ID --type="merge" -p "{\"spec\":{\"clusterTemplate\":{\"gitRepoTemplate\":{\"apiSecretRef\":{\"name\":\"gitlab-com\"},\"path\":\"${GITLAB_USERNAME}\",\"repoName\":\"{{ .Name }}\"},\"tenantRef\":{}}}}"
This patch is needed because of the new feature implemented in the Operator in PR #110 "Add cluster template to tenant". It will be added to the API in Issue #89 "Expose Cluster Template Feature in Tenant Objects".
-
Retrieve the registered Tenants via API and directly on the cluster
curl -H "$LIEUTENANT_AUTH" "http://${LIEUTENANT_URL}/tenants" kubectl -n lieutenant get tenant kubectl -n lieutenant get gitrepo
-
Register a Lieutenant Cluster via the API
CLUSTER_ID=$(curl -s -H "$LIEUTENANT_AUTH" -H "Content-Type: application/json" -X POST --data "{ \"tenant\": \"${TENANT_ID}\", \"displayName\": \"My first Project Syn cluster\", \"facts\": { \"cloud\": \"local\", \"distribution\": \"k3s\", \"region\": \"local\" }, \"gitRepo\": { \"url\": \"ssh://git@${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/cluster-gitops1.git\" } }" "http://${LIEUTENANT_URL}/clusters" | jq -r ".id") echo $CLUSTER_ID echo https://${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/cluster-gitops1
If everything went well, the Lieutenant Operator created a new git repository under ${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/cluster-gitops1 which will be used to store the generated catalog of deployment files.
-
Retrieve the registered Clusters via API and directly on the cluster
curl -H "$LIEUTENANT_AUTH" "http://${LIEUTENANT_URL}/clusters" kubectl -n lieutenant get cluster kubectl -n lieutenant get gitrepo
Kickstart Commodore
Commodore is the configuration generation tool. It will be configured to generate configuration for your Lieutenant cluster $CLUSTER_ID
generated above. With all the information available in Lieutenant, Commodore is able to figure out what to actually compile for the cluster in question and where to Git push the compiled catalog to.
Before continuing with this section, make sure that everything went well with the installation and configuration of Lieutenant as Commodore relies on having a working instance of it.
Run Commodore
The easiest way of executing Commodore is by using the container image provided by Project Syn: docker.io/projectsyn/commodore. We run the image directly in the local k3s
or docker-desktop
instance so that there is no need for having another container runtime installed.
Execute the following command which will start the properly configured Commodore container inside your local k3s
or docker-desktop
instance.
Replace MYSSHKEYPATH
with the path to your SSH key file, for example ~/.ssh/id_rsa
. This SSH key will be used to push the generated configuration catalog to the Git repository managed by Lieutenant.
export COMMODORE_SSH_PRIVATE_KEY=MYSSHKEYPATH
kubectl -n lieutenant run commodore-shell \
--image=docker.io/projectsyn/commodore:v0.4.0 \
--env=COMMODORE_API_URL="http://${LIEUTENANT_URL}/" \
--env=COMMODORE_API_TOKEN=${LIEUTENANT_TOKEN} \
--env=SSH_PRIVATE_KEY="$(cat ${COMMODORE_SSH_PRIVATE_KEY})" \
--env=CLUSTER_ID=${CLUSTER_ID} \
--env=GITLAB_ENDPOINT=${GITLAB_ENDPOINT} \
--tty --stdin --restart=Never --rm --wait \
--image-pull-policy=Always \
--command \
-- /usr/local/bin/entrypoint.sh bash
If your SSH key is protected by a passphrase (hopefully so!) no command prompt will be displayed and it will look like it halted at If you don’t see a command prompt, try pressing enter
. Don’t just press "enter" but type your SSH key passphrase (an ssh-agent
is started in the container’s entrypoint) and press "enter" after that.
When there is no passphrase on your SSH key, the command prompt should directly show up.
Now execute (inside the container):
On macOS
|
ssh-keyscan ${GITLAB_ENDPOINT} >> /app/.ssh/known_hosts
commodore catalog compile $CLUSTER_ID --push
The output will look like this:
Cleaning working tree
Updating global config...
Updating customer config...
Discovering components...
Fetching components...
Updating Kapitan target...
Updating cluster catalog...
> Reference at 'refs/heads/master' does not exist, creating initial commit for catalog
Updating Jsonnet libraries...
Cleaning catalog repository...
> Converting old-style catalog
Updating Kapitan secret references...
Compiling catalog...
...
> Commiting changes...
> Pushing catalog to remote...
Catalog compiled! 🎉
You now have your first Commodore compiled catalog available under catalog/
and pushed to GitLab to the cluster catalog repository.
To see what was just generated, browse to ${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/cluster-gitops1 (or do a find catalog/
) to see the Git commit (and Git push) Commodore created and all the generated Kubernetes objects. These objects will then actually be applied to the cluster by Argo CD (we’ve not installed Argo CD in this guide).
This guide uses github.com/projectsyn/getting-started-commodore-defaults/ as the global common configuration repository. If you want to use your own, adapt the globalGitRepoURL in the Tenant spec or update the Operator configuration env var DEFAULT_GLOBAL_GIT_REPO_URL .
|
Now exit the Commodore container by typing exit
. This also deletes the Pod on the local k3s
or docker-desktop
instance.
Kickstart Steward
With Lieutenant running and having a compiled cluster catalog by Commodore available, it’s now time to enable Syn on the local k3s
or docker-desktop
instance and get it GitOps managed. This is the job of Steward, the in-cluster agent of Project Syn.
The installation of Steward is done via a cluster specific install URL which contains a one-time bootstrap token. This token is only valid once and only for 30 minutes after cluster registration.
-
Check the validity of the bootstrap token
kubectl -n lieutenant get cluster ${CLUSTER_ID} -o jsonpath="{.status.bootstrapToken.tokenValid}" kubectl -n lieutenant get cluster ${CLUSTER_ID} -o jsonpath="{.status.bootstrapToken.validUntil}"
If this doesn’t return
true
, have a look at the tip below about how to reset the token. -
Retrieve the Steward install URL
export STEWARD_INSTALL=$(curl -H "$LIEUTENANT_AUTH" -s "http://${LIEUTENANT_URL}/clusters/${CLUSTER_ID}" | jq -r ".installURL") echo $STEWARD_INSTALL
-
Install Steward in the local
k3s
ordocker-desktop
instancekubectl apply -f $STEWARD_INSTALL
-
Check the validity of the bootstrap token
kubectl -n lieutenant get cluster ${CLUSTER_ID} -o jsonpath="{.status.bootstrapToken.tokenValid}"
This command should return
false
. -
Check that Steward is running and that Argo CD Pods are appearing
kubectl -n syn get pod
This should list 5 Pods, maybe still in
ContainerCreating
. -
Check that an SSH deploy key has been added to the catalog repository by browsing to ${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/cluster-gitops1/-/settings/repository. Click on
Expand
next toDeploy Keys
, there you should find one deploy key calledsteward
. -
Check that Argo CD was able to sync the changes
kubectl -n syn get app root -o jsonpath="{.status.sync.status}"
This command should return
Synced
. -
Retrieve the admin password for Argo CD
kubectl -n syn get secret steward -o json | jq -r .data.token | base64 --decode
-
Now you can access Argo CD by forwarding the port and opening it in your browser with localhost:8443. Login with the username
admin
and the password retrieved in the previous step.kubectl -n syn port-forward svc/argocd-server 8443:443
With these steps, the local k3s
or docker-desktop
instance is now Syn enabled, has Argo CD running and automatically syncs the manifests found in the cluster catalog Git repository which was generated by Commodore and is stored in GitLab under ${GITLAB_ENDPOINT}/${GITLAB_USERNAME}/cluster-gitops1/.
If you want or need to reset the bootstrap token, this is the way to go:
Get the Kubernetes API URL with
|
Cleaning Up
Once you’ve gone through all these steps, you can cleanup all generated stuff using the following steps:
-
Delete the
Cluster
objectkubectl -n lieutenant delete cluster ${CLUSTER_ID}
This will also delete the associated
GitRepo
object and with that the cluster configuration file in the tenant configuration repository and the cluster catalog Git repository on GitLab. -
Delete the
Tenant
objectkubectl -n lieutenant delete tenant ${TENANT_ID}
This will also delete the associated
GitRepo
object and with that the tenant configuration Git repository on GitLab. -
Delete the k3d cluster
k3d cluster delete projectsyn