This is the multi-page printable view of this section. Click here to print.
We officially only support the installation of Capsule using the Helm chart. The chart itself handles the installation/upgrade of the required CustomResourceDefinitions. The following Artifact Hub repositories are official:
Perform the following steps to install the Capsule operator:
Add repository:
helm repo add projectcapsule https://projectcapsule.github.io/charts
Install Capsule:
helm install capsule projectcapsule/capsule --version 0.14.6 -n capsule-system --create-namespace
or (OCI)
helm install capsule oci://ghcr.io/projectcapsule/charts/capsule --version 0.14.6 -n capsule-system --create-namespace
Show the status:
helm status capsule -n capsule-system
Upgrade the Chart
helm upgrade capsule projectcapsule/capsule -n capsule-system
or (OCI)
helm upgrade capsule oci://ghcr.io/projectcapsule/charts/capsule --version 0.14.6
Uninstall the Chart
helm uninstall capsule -n capsule-system
Here are some key considerations to keep in mind when installing Capsule. Also check out the Best Practices for more information.
For large clusters you might need to consider adjusting values for the Capsule controller.
In order to handle a large number of tenants and resources, you may need to increase the QPS and Burst values for the Capsule controller. This avoids the controller being throttled by the Kubernetes API server (Client Rate limited). You can set the following values in the Helm chart:
manager:
options:
clientConnectionQPS: 400
clientConnectionBurst: 200
Define the number of workers for the Capsule controller, which translates into the number of concurrent reconciles:
manager:
options:
workers: 4
The more resources you have in your cluster, the longer it will take for the Capsule controller to sync its cache. You can adjust the cache sync period to a higher value to reduce the load on the API server:
manager:
options:
cacheSyncTimeout: "10m"
In high pressure environments leader election may fail due to the default timeout values. You can adjust the leader election timeout values to avoid this issue:
E0707 08:38:18.319041 1 leaderelection.go:452] "Error retrieving lease lock"
err="Get \"https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/capsule-
system/leases/42c733ea.clastix.capsule.io?timeout=5s\": net/http: request canceled
(Client.Timeout exceeded while awaiting headers)" lock="capsule-
system/42c733ea.clastix.capsule.io"
I0707 08:38:18.442700 1 leaderelection.go:299] "Failed to renew lease"
Tune leader election with manager.options.leaderElection.leaseDuration, manager.options.leaderElection.renewDeadline, and manager.options.leaderElection.retryPeriod. Increasing these values makes Capsule more tolerant of slow or overloaded Kubernetes API servers; for example, raising renewDeadline also raises the leader-election client request timeout because controller-runtime uses roughly half of that value. The tradeoff is slower failover: if the active controller really dies, standby replicas will wait longer before taking leadership. Keep the ordering valid: leaseDuration should be greater than renewDeadline, and renewDeadline should be greater than retryPeriod.
manager:
options:
leaderElection:
leaseDuration: "60s"
renewDeadline: "40s"
retryPeriod: "5s"
Worst-case leader failover is slower, around 60s, if the active pod really dies. Keep manager.options.leaderElection.leaseDuration > manager.options.leaderElection.renewDeadline > manager.options.leaderElection.retryPeriod.
With APF enabled, the Capsule controller will be subject to the APF configuration of the cluster. If you are running a large cluster with many tenants, you may need to adjust the APF configuration to ensure that the Capsule controller has sufficient resources to operate effectively. For more information on APF, see Kubernetes API Priority and Fairness.
We provide a built-in APF configuration for the Capsule controller, which provides API priority for all resources managed by Capsule. This configuration is applied automatically when you install Capsule. To enable the built-in APF configuration, set the following value in the Helm chart:
# Manager Options
manager:
apiPriorityAndFairness:
# -- Change to `true` if you want to insulate the API calls made by Capsule admission controller activities.
# This will help ensure Capsule stability in busy clusters.
# Ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/
enabled: true
# -- Only the first matching FlowSchema for a given request matters. If multiple FlowSchemas match a single inbound request, it will be assigned based on the one with the highest matchingPrecedence.
# Ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#flowschema
matchingPrecedence: 900
# -- Priority level configuration.
# The block is directly forwarded into the priorityLevelConfiguration, so you can use whatever specification you want.
# ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#prioritylevelconfiguration
priorityLevelConfigurationSpec:
type: Limited
limited:
nominalConcurrencyShares: 100
limitResponse:
type: Queue
queuing:
queues: 64
handSize: 6
queueLengthLimit: 100
0.13.0 of capsule before enabling strict mode. As it requires fields which are newly added with version 0.13.0.By default, the Capsule controller runs with the ClusterRole cluster-admin, which provides full access to the cluster. This is because the controller must be able to grant RoleBindings on a per-namespace basis: Kubernetes prevents privilege escalation through RBAC, so a RoleBinding referencing a ClusterRole can only be created by a subject that either already holds all permissions of that ClusterRole or holds the bind verb on it. For production environments we recommend configuring stricter RBAC permissions for the Capsule controller by setting the following value in the Helm chart:
manager:
rbac:
strict: true
This replaces the cluster-admin binding with a ClusterRole that grants the controller only the permissions required for its own operation (manager.rbac.minimal is a deprecated alias for this option; both flags enable the same behavior). To satisfy the escalation prevention described above, this ClusterRole contains the narrow bind verb on the ClusterRoles listed in manager.rbac.bindableClusterRoles (default: admin) plus the ClusterRoles managed by Capsule itself (the manager.options.rbac provisioner, deleter, administrationClusterRoles and promotionClusterRoles), which are always bindable. With default values this renders as:
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["clusterroles"]
resourceNames: ["admin", "capsule-namespace-deleter", "capsule-namespace-provisioner"]
verbs: ["bind"]
Tenant owner RoleBindings for the default ClusterRoles therefore work out of the box, without the controller inheriting all of admin’s permissions.
On older chart versions strict mode is only enabled via manager.rbac.minimal: true (manager.rbac.strict had no effect there), and the controller ClusterRole includes neither the bind rule above nor create on NetworkPolicies, which GlobalTenantResource propagation requires. Add both via Helm values:
manager:
rbac:
minimal: true
clusterRole:
extraResources:
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["clusterroles"]
resourceNames: ["admin", "capsule-namespace-provisioner", "capsule-namespace-deleter"]
verbs: ["bind"]
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
Or aggregate the full admin ClusterRole to the controller — note that the controller then holds all of admin’s permissions:
kubectl label clusterrole admin projectcapsule.dev/aggregate-to-controller=true
Any further ClusterRole that is assigned to Tenant owners or used for additional RoleBindings needs the same treatment. This applies only to ClusterRoles that are not managed by Capsule (see Configuration). The narrowest option is adding it to manager.rbac.bindableClusterRoles — note that the list replaces the default, so keep admin in it:
manager:
rbac:
strict: true
bindableClusterRoles:
- admin
- prometheus-servicemonitors-viewer
Alternatively, you can aggregate the permissions of further ClusterRoles into the controller. The ClusterRole for the controller aggregates all ClusterRoles carrying one of the following labels:
projectcapsule.dev/aggregate-to-controller: "true"projectcapsule.dev/aggregate-to-controller-instance: {{ .Release.Name }}kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: prometheus-servicemonitors-viewer
labels:
projectcapsule.dev/aggregate-to-controller: "true"
rules:
...
This is broader than bind — the controller then actually holds these permissions. It is required whenever the controller must exercise the permissions itself, for example when replicating resources via GlobalTenantResources without impersonation.
You can also directly grant arbitrary additional permissions via Helm values:
manager:
rbac:
strict: true
clusterRole:
extraResources:
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch", "update", "patch"]
If you are missing permissions you will see an error status for the respective tenants reflecting
kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE
green Active 2 False cannot sync rolebindings items: rolebindings.rbac.authorization.k8s.io "capsule:managed:658936e7f2a30e35" is forbidden: user "system:serviceaccount:capsule-system:capsule" (groups=["system:serviceaccounts" "system:serviceaccounts:capsule-system" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:... 5s
Before you enable strict mode, you must implement the required permissions for your use case. Depending on which features you are using, you may need to take manual action, for example:
TenantResources impersonation ServiceAccountGlobalTenantResources to use impersonationWhile Capsule provides a robust framework for managing multi-tenancy in Kubernetes, it does not include built-in admission policies for enforcing specific security or operational standards for all possible aspects of a Kubernetes cluster. We provide additional policy recommendations here.
By default, Capsule delegates its certificate management to cert-manager. This is the recommended way to manage the TLS certificates for Capsule. However, you can also use Capsule’s built-in TLS reconciler to manage the certificates. This is not recommended for production environments. To enable the TLS reconciler, use the following values:
certManager:
generateCertificates: false
tls:
enableController: true
create: true
Capsule makes use of webhooks for admission control. Ensure that your cluster supports webhooks and that they are properly configured. The webhooks are automatically created by Capsule during installation. However, some of these webhooks will cause problems when Capsule is not running (this is especially problematic in single-node clusters). Here are the webhooks you need to watch out for.
Generally, we recommend using matchConditions for all webhooks to avoid problems when Capsule is not running. You should exclude your system-critical components from the Capsule webhooks. For namespaced resources (pods, services, etc.) the webhooks select only namespaces that are part of a Capsule Tenant. If your system-critical components are not part of a Capsule Tenant, they will not be affected by the webhooks. However, if you have system-critical components that are part of a Capsule Tenant, you should exclude them from the Capsule webhooks by using matchConditions as well, or add more specific namespaceSelectors/objectSelectors to exclude them. This can also improve performance.
It’s strongly advised to exclude system-critical components from the Capsule webhooks. This includes components like kube-system, kube-public, and kube-node-lease namespaces, as well as any other namespaces that are critical for the operation of your cluster. You can use matchConditions to exclude these namespaces from the Capsule webhooks.
webhooks:
matchConditions:
- name: exclude-control-plane
expression: '!(request.userInfo.username in ["system:apiserver", "system:kube-controller-manager", "system:kube-scheduler"]) && !("system:nodes" in request.userInfo.groups) && !("system:serviceaccounts:kube-system" in request.userInfo.groups)'
There is a webhook which catches interactions with the Node resource. This webhook is mainly relevant when you make use of Node metadata. In most other cases, it will only cause problems. By default, the webhook is disabled, but you can enable it by setting the following value:
webhooks:
hooks:
nodes:
enabled: true
Or you could at least consider to set the failure policy to Ignore, if you don’t want to disrupt critical nodes:
webhooks:
hooks:
nodes:
failurePolicy: Ignore
If you still want to use the feature, you could exclude the kube-system namespace (or any other namespace you want to exclude) from the webhook by setting the following value:
webhooks:
hooks:
nodes:
matchConditions:
- name: 'exclude-kubelet-requests'
expression: '!("system:nodes" in request.userInfo.groups)'
- name: 'exclude-kube-system'
expression: '!("system:serviceaccounts:kube-system" in request.userInfo.groups)'
Namespaces are the most important resource in Capsule. The Namespace webhook is responsible for enforcing the Capsule Tenant boundaries. It is enabled by default and should not be disabled. However, you may change the matchConditions to exclude certain namespaces from the Capsule Tenant boundaries. For example, you can exclude the kube-system namespace by setting the following value:
webhooks:
hooks:
namespaces:
matchConditions:
- name: 'exclude-kube-system'
expression: '!("system:serviceaccounts:kube-system" in request.userInfo.groups)'
By default resources with the following values are protected by a webhook to be changed by [Capsule Users]:
webhooks:
hooks:
managed:
objectSelector:
matchExpressions:
- key: "projectcapsule.dev/created-by"
operator: In
values:
- "controller"
- "resources"
- key: "projectcapsule.dev/managed-by"
operator: In
values:
- "controller"
There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource.
Visit the ArgoCD Integration for more options to integrate Capsule with ArgoCD.
Manifests to get you started with ArgoCD. For ArgoCD you might need to skip the validation of the CapsuleConfiguration resources, otherwise there might be errors on the first install:
Validate=false option is required for the CapsuleConfiguration resource, because ArgoCD tries to validate the resource before the Capsule CRDs are installed via our CRD Lifecycle hook. Upstream Issue. This has mainly been observed in ArgoCD Applications using Service-Side Diff/Apply.manager:
options:
annotations:
argocd.argoproj.io/sync-options: "Validate=false,SkipDryRunOnMissingResource=true"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: capsule
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: system
source:
repoURL: ghcr.io/projectcapsule/charts
targetRevision: 0.14.6
chart: capsule
helm:
valuesObject:
crds:
install: true
manager:
options:
annotations:
argocd.argoproj.io/sync-options: "Validate=false,SkipDryRunOnMissingResource=true"
capsuleConfiguration: default
ignoreUserGroups:
- oidc:administators
users:
- kind: Group
name: oidc:kubernetes-users
- kind: Group
name: system:serviceaccounts:tenants-system
monitoring:
dashboards:
enabled: true
serviceMonitor:
enabled: true
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
destination:
server: https://kubernetes.default.svc
namespace: capsule-system
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
- RespectIgnoreDifferences=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
---
apiVersion: v1
kind: Secret
metadata:
name: capsule-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
url: ghcr.io/projectcapsule/charts
name: capsule
project: system
type: helm
enableOCI: "true"
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: capsule
namespace: flux-system
spec:
serviceAccountName: kustomize-controller
targetNamespace: "capsule-system"
interval: 10m
releaseName: "capsule"
chart:
spec:
chart: capsule
version: "0.14.6"
sourceRef:
kind: HelmRepository
name: capsule
interval: 24h
install:
createNamespace: true
upgrade:
remediation:
remediateLastFailure: true
driftDetection:
mode: enabled
values:
crds:
install: true
manager:
options:
capsuleConfiguration: default
ignoreUserGroups:
- oidc:administators
users:
- kind: Group
name: oidc:kubernetes-users
- kind: Group
name: system:serviceaccounts:tenants-system
monitoring:
dashboards:
enabled: true
serviceMonitor:
enabled: true
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: capsule
namespace: flux-system
spec:
type: "oci"
interval: 12h0m0s
url: oci://ghcr.io/projectcapsule/charts
To verify artifacts you need to have cosign installed. This guide assumes you are using v2.x of cosign. All of the signatures are created using keyless signing. You can set the environment variable COSIGN_REPOSITORY to point to this repository. For example:
# Docker Image
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule
# Helm Chart
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule
To verify the signature of the docker image, run the following command.
COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign verify ghcr.io/projectcapsule/capsule:<release_tag> \
--certificate-identity-regexp="https://github.com/projectcapsule/capsule/.github/workflows/docker-publish.yml@refs/tags/*" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" | jq
To verify the signature of the helm image, run the following command.
COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign verify ghcr.io/projectcapsule/charts/capsule:<release_tag> \
--certificate-identity-regexp="https://github.com/projectcapsule/capsule/.github/workflows/helm-publish.yml@refs/tags/*" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" | jq
Capsule creates and attests to the provenance of its builds using the SLSA standard and meets the SLSA Level 3 specification. The attested provenance may be verified using the cosign tool.
Verify the provenance of the docker image.
cosign verify-attestation --type slsaprovenance \
--certificate-identity-regexp="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
ghcr.io/projectcapsule/capsule:0.14.6 | jq .payload -r | base64 --decode | jq
cosign verify-attestation --type slsaprovenance \
--certificate-identity-regexp="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
ghcr.io/projectcapsule/charts/capsule:0.14.6 | jq .payload -r | base64 --decode | jq
An SBOM (Software Bill of Materials) in CycloneDX JSON format is published for each release, including pre-releases. You can set the environment variable COSIGN_REPOSITORY to point to this repository. For example:
# Docker Image
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule
# Helm Chart
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule
To inspect the SBOM of the docker image, run the following command.
COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule cosign download sbom ghcr.io/projectcapsule/capsule:0.14.6
To inspect the SBOM of the helm image, run the following command.
COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign download sbom ghcr.io/projectcapsule/charts/capsule:0.14.6
The Kubernetes compatibility is announced for each Release. Generally we are up to date with the latest upstream Kubernetes Version. Note that the Capsule project offers support only for the latest minor version of Kubernetes. Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors.
The configuration for the capsule controller is done via its dedicated configuration Custom Resource. You can explain the configuration options and how to use them:
The configuration for Capsule is done via its dedicated configuration Custom Resource. You can explain the configuration options and how to use them:
kubectl explain capsuleConfiguration.spec
administratorsThese entities are automatically owners for all existing tenants. Meaning they can add namespaces to any tenant. However they must be specific by using the capsule label for interacting with namespaces. Because if that label is not defined, it’s assumed that namespace interaction was not targeted towards a tenant and will therefore be ignored by capsule. May also be handy in GitOps scenarios where certain service accounts need to be able to manage namespaces for all tenants.
manager:
options:
administrators:
- kind: User
name: admin-user
usersThese entities are automatically owners for all existing tenants. Meaning they can add namespaces to any tenant. However they must be specific by using the capsule label for interacting with namespaces. Because if that label is not defined, it’s assumed that namespace interaction was not targeted towards a tenant and will therefore be ignored by capsule. May also be handy in GitOps scenarios where certain service accounts need to be able to manage namespaces for all tenants.
manager:
options:
users:
- kind: User
name: owner-user
- kind: Group
name: projectcapsule.dev
ignoreUserWithGroupsDefine groups which when found in the request of a user will be ignored by the Capsule. This might be useful if you have one group where all the users are in, but you want to separate administrators from normal users with additional groups.
manager:
options:
ignoreUserWithGroups:
- company:org:administrators
enableTLSReconcilerToggles the TLS reconciler, the controller that is able to generate CA and certificates for the webhooks when not using an already provided CA and certificate, or when these are managed externally with Vault, or cert-manager.
tls:
enableController: true
forceTenantPrefixEnforces the Tenant owner, during Namespace creation, to name it using the selected Tenant name as prefix, separated by a dash. This is useful to avoid Namespace name collision in a public CaaS environment.
manager:
options:
forceTenantPrefix: true
nodeMetadataAllows to set the forbidden metadata for the worker nodes that could be patched by a Tenant. This applies only if the Tenant has an active NodeSelector, and the Owner have right to patch their nodes.
manager:
options:
nodeMetadata:
forbiddenLabels:
denied:
- "node-role.kubernetes.io/*"
deniedRegex: ""
forbiddenAnnotations:
denied:
- "node.alpha.kubernetes.io/*"
deniedRegex: ""
overridesAllows to set different name rather than the canonical one for the Capsule configuration objects, such as webhook secret or configurations.
protectedNamespaceRegexDisallow creation of namespaces, whose name matches this regexp
manager:
options:
protectedNamespaceRegex: "^(kube|default|capsule|admin|system|com|org|local|localhost|io)$"
allowServiceAccountPromotionServiceAccounts within tenant namespaces can be promoted to owners of the given tenant this can be achieved by labeling the serviceaccount and then they are considered owners. This can only be done by other owners of the tenant. However ServiceAccounts which have been promoted to owner can not promote further serviceAccounts.
manager:
options:
allowServiceAccountPromotion: true
cacheInvalidationThe reconcile periode caches are invalidated. Invalidation is already attempted when resources change, however in certain scenarios it might be necessary to do out of order cache invalidations to ensure proper garbage collection of resources.
manager:
options:
cacheInvalidation: 0h30m0s
rbacDefine configurations for the RBAC which is being managed and applied by Capsule.
manager:
options:
rbac:
# -- The ClusterRoles applied for Administrators
administrationClusterRoles:
- capsule-namespace-deleter
# -- The ClusterRoles applied for ServiceAccounts which had owner Promotion
promotionClusterRoles:
- capsule-namespace-provisioner
- capsule-namespace-deleter
# -- Name for the ClusterRole required to grant Namespace Deletion permissions.
deleter: capsule-namespace-deleter
# -- Name for the ClusterRole required to grant Namespace Provision permissions.
provisioner: capsule-namespace-provisioner
impersonationReplications impersonate a ServiceAccount, falling back to the controller ServiceAccount when none is resolved. tenantDefaultServiceAccount is set to default for TenantResource; GlobalTenantResource has no default and requires both globalDefaultServiceAccount and globalDefaultServiceAccountNamespace. Within this configuration you can define properties such as the endpoint of the kube-apiserver and if service account promotion should be allowed for this client. By default the https://kubernetes.default.svc endpoint is used.
manager:
options:
impersonation:
# Kubernetes API Endpoint to use for the operations
endpoint: "https://capsule-proxy.capsule-system.svc:8081"
# Toggles if TLS verification for the endpoint is performed or not
skipTlsVerify: false
# Key in the secret that holds the CA certificate (e.g., "ca.crt")
caSecretKey: "ca.crt"
# Name of the secret containing the CA certificate
caSecretName: "capsule-proxy-tls"
# Namespace where the CA certificate secret is located
caSecretNamespace: "capsule-system"
# Default ServiceAccount for global resources (GlobalTenantResource) [Cluster Scope]
# When defined, users are required to use this ServiceAccount anywhere in the cluster
# unless they explicitly provide their own. Once this is set, Capsule will add this ServiceAccount
# for all GlobalTenantResources, if they don't already have a ServiceAccount defined.
globalDefaultServiceAccount: "capsule-global-sa"
# Namespace of the for the ServiceAccount provided by the globalDefaultServiceAccount property
globalDefaultServiceAccountNamespace: "tenant-system"
# Default ServiceAccount for tenant resources (TenantResource) [Namespaced Scope]
# Applied to every TenantResource which doesn't define its own, in that resource's
# own Namespace.
tenantDefaultServiceAccount: "default"
admissionConfiguration for the dynamic admission webhooks used by Capsule for mutating and validating requests. The settings are used from the static webhook configurations created during installation of Capsule and abstracted by the helm chart
manager:
options:
admission:
mutating:
client:
caBundle: cert
url: https://172.24.52.212:9443
name: capsule-dynamic
validating:
client:
caBundle: cert
url: https://172.24.52.212:9443
name: capsule-dynamic
Depending on the version of the Capsule Controller, the configuration options may vary. You can view the options for the latest version of the Capsule Controller or by executing the controller locally:
$ go run ./cmd/controller/ --zap-log-level 7 -h
--cache-sync-timeout duration The timeout used when waiting for controller cache synchronization. If unset or 0, the controller-runtime default is used.
--client-connection-burst int32 Burst to use for interacting with kubernetes apiserver. (default 30)
--client-connection-qps float32 QPS to use for interacting with kubernetes apiserver. (default 20)
--configuration-name string The CapsuleConfiguration resource name to use (default "default")
--enable-http2 If set, HTTP/2 will be enabled for the metrics and webhook servers
--enable-leader-election Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
--enable-pprof Enables Pprof endpoint for profiling (not recommend in production)
--metrics-addr string The address the metric endpoint binds to. (default ":8080")
--metrics-cert-key string The name of the metrics server key file. (default "tls.key")
--metrics-cert-name string The name of the metrics server certificate file. (default "tls.crt")
--metrics-cert-path string The directory that contains the metrics server certificate.
--metrics-secure If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.
--version Print the Capsule version and exit
--webhook-cert-key string The name of the webhook key file. (default "tls.key")
--webhook-cert-name string The name of the webhook certificate file. (default "tls.crt")
--webhook-cert-path string The directory that contains the webhook certificate. (default "/tmp/k8s-webhook-server/serving-certs")
--webhook-port int The port the webhook server binds to. (default 9443)
--workers int MaxConcurrentReconciles is the maximum number of concurrent Reconciles which can be run. (default 1)
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
Define additional options in the values.yaml when installing via Helm:
manager:
extraArgs:
- "--enable-leader-election=true"
Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by userGroups option in the CapsuleConfiguration, which defaults to projectcapsule.dev.
In the following guide, we’ll use Keycloak an Open Source Identity and Access Management server capable to authenticate users via OIDC and release JWT tokens as proof of authentication.
Configure Keycloak as OIDC server:
projectcapsule.devprojectcapsule.devkubernetes (Public)kubernetes-auth (Confidential (Client Secret))For the kubernetes client, create protocol mappers called groups and audience If everything is done correctly, now you should be able to authenticate in Keycloak and see user groups in JWT tokens. Use the following snippet to authenticate in Keycloak as alice user:
$ KEYCLOAK=sso.clastix.io
$ REALM=kubernetes-auth
$ OIDC_ISSUER=${KEYCLOAK}/realms/${REALM}
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/token \
-d grant_type=password \
-d response_type=id_token \
-d scope=openid \
-d client_id=${OIDC_CLIENT_ID} \
-d client_secret=${OIDC_CLIENT_SECRET} \
-d username=${USERNAME} \
-d password=${PASSWORD} | jq
The result will include an ACCESS_TOKEN, a REFRESH_TOKEN, and an ID_TOKEN. The access-token can generally be disregarded for Kubernetes. It would be used if the identity provider was managing roles and permissions for the users but that is done in Kubernetes itself with RBAC. The id-token is short lived while the refresh-token has longer expiration. The refresh-token is used to fetch a new id-token when the id-token expires.
{
"access_token":"ACCESS_TOKEN",
"refresh_token":"REFRESH_TOKEN",
"id_token": "ID_TOKEN",
"token_type":"bearer",
"scope": "openid groups profile email"
}
To introspect the ID_TOKEN token run:
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/introspect \
-d token=${ID_TOKEN} \
--user kubernetes-auth:${OIDC_CLIENT_SECRET} | jq
The result will be like the following:
{
"exp": 1601323086,
"iat": 1601322186,
"aud": "kubernetes",
"typ": "ID",
"azp": "kubernetes",
"preferred_username": "alice",
"email_verified": false,
"acr": "1",
"groups": [
"capsule.clastix.io"
],
"client_id": "kubernetes",
"username": "alice",
"active": true
}
Configuring Kubernetes for OIDC Authentication requires adding several parameters to the API Server. Please, refer to the documentation for details and examples. Most likely, your kube-apiserver.yaml manifest will looks like the following:
The configuration file approach allows you to configure multiple JWT authenticators, each with a unique issuer.url and issuer.discoveryURL. The configuration file even allows you to specify CEL expressions to map claims to user attributes, and to validate claims and user information.
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://${OIDC_ISSUER}
audiences:
- kubernetes
- kubernetes-auth
audienceMatchPolicy: MatchAny
claimMappings:
username:
claim: 'email'
prefix: ""
groups:
claim: 'groups'
prefix: ""
certificateAuthority: <PEM encoded CA certificates>
This file must be present and consistent across all kube-apiserver instances in the cluster. Add the following flag to the kube-apiserver manifest:
spec:
containers:
- command:
- kube-apiserver
...
- --authentication-configuration-file=/etc/kubernetes/authentication/authentication.yaml
spec:
containers:
- command:
- kube-apiserver
...
- --oidc-issuer-url=https://${OIDC_ISSUER}
- --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
- --oidc-client-id=kubernetes
- --oidc-username-claim=preferred_username
- --oidc-groups-claim=groups
- --oidc-username-prefix=-
As reference, here is an example of a KinD configuration for OIDC Authentication, which can be useful for local testing:
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
extraArgs:
oidc-issuer-url: https://${OIDC_ISSUER}
oidc-username-claim: preferred_username
oidc-client-id: kubernetes
oidc-username-prefix: "keycloak:"
oidc-groups-claim: groups
oidc-groups-prefix: "keycloak:"
enable-admission-plugins: PodNodeSelector
There are two options to use kubectl with OIDC:
Plugin
One way to use OIDC authentication is the use of a kubectl plugin. The Kubelogin Plugin for kubectl simplifies the process of obtaining an OIDC token and configuring kubectl to use it. Follow the link to obtain installation instructions.
kubectl oidc-login setup \
--oidc-issuer-url=https://${OIDC_ISSUER} \
--oidc-client-id=kubernetes-auth \
--oidc-client-secret=${OIDC_CLIENT_SECRET}
Manual
To use the OIDC Authenticator, add an oidc user entry to your kubeconfig file:
$ kubectl config set-credentials oidc \
--auth-provider=oidc \
--auth-provider-arg=idp-issuer-url=https://${OIDC_ISSUER} \
--auth-provider-arg=idp-certificate-authority=/path/to/ca.crt \
--auth-provider-arg=client-id=kubernetes-auth \
--auth-provider-arg=client-secret=${OIDC_CLIENT_SECRET} \
--auth-provider-arg=refresh-token=${REFRESH_TOKEN} \
--auth-provider-arg=id-token=${ID_TOKEN} \
--auth-provider-arg=extra-scopes=groups
To use the --token option:
$ kubectl config set-credentials oidc --token=${ID_TOKEN}
Point the kubectl to the URL where the Kubernetes APIs Server is reachable:
$ kubectl config set-cluster mycluster \
--server=https://kube.projectcapsule.io:6443 \
--certificate-authority=~/.kube/ca.crt
If your APIs Server is reachable through the capsule-proxy, make sure to use the URL of the capsule-proxy.
Create a new context for the OIDC authenticated users:
$ kubectl config set-context alice-oidc@mycluster \
--cluster=mycluster \
--user=oidc
As user alice, you should be able to use kubectl to create some namespaces:
$ kubectl --context alice-oidc@mycluster create namespace wind-production
$ kubectl --context alice-oidc@mycluster create namespace wind-development
$ kubectl --context alice-oidc@mycluster create namespace water-marketing
Warning: once your ID_TOKEN expires, the kubectl OIDC Authenticator will attempt to refresh automatically your ID_TOKEN using the REFRESH_TOKEN. In case the OIDC uses a self signed CA certificate, make sure to specify it with the idp-certificate-authority option in your kubeconfig file, otherwise you’ll not able to refresh the tokens.
Capsule is a Kubernetes multi-tenancy operator that enables secure namespace-as-a-service in Kubernetes clusters. When combined with OpenShift’s robust security model, it provides an excellent platform for multi-tenant environments.
This guide demonstrates how to deploy Capsule and Capsule Proxy on OpenShift using the nonroot-v2 and restricted-v2 SecurityContextConstraint (SCC), ensuring tenant owners operate within OpenShift’s security boundaries.
While OpenShift can already be configured for multi-tenancy (for example with Kyverno), Capsule takes it a step further and makes it easier to manage.
When people think of a multi-tenant Kubernetes cluster, they often expect one or two namespaces with few privileges. Capsule, however, is different. As a tenant owner, you can create as many namespaces as you want. RBAC is much easier because Capsule handles it, making it less error-prone. Resource quotas are not set per namespace but are spread across the whole tenant, simplifying management. Capsule Proxy also solves RBAC issues when listing cluster-wide resources. Furthermore, some operators can be installed inside a tenant by using the Capsule Proxy: add the service account as a tenant owner and set the KUBERNETES_SERVICE_HOST environment variable of the operator deployment to the Capsule Proxy URL. The operator then behaves as if it has cluster-admin access, while remaining fully confined to the tenant.
Before starting, ensure you have:
kubectl CLI configuredThe following limitations are known when using OpenShift with Capsule:
kubectl.login token from the OpenShift GUI, the server address will always point to the Kubernetes API instead of the Capsule Proxy, so the --server flag has to be adjusted manually. Browser-based oc login --web against the Capsule Proxy works without this workaround when public paths are configured (see Capsule Proxy). An RFE has been filed with Red Hat to make this URL configurable (RFE-7592). If you have a support contract with Red Hat, consider opening a support request (SR) asking for this feature. The more requests there are, the higher the priority.By default, OpenShift includes a self-provisioner role and ClusterRoleBinding that allows all users to create namespaces. Capsule requires this to be removed. See the Red Hat documentation for details.
Remove the subjects from the ClusterRoleBinding:
kubectl patch clusterrolebinding.rbac self-provisioners -p '{"subjects": null}'
Also set autoupdate to false so the ClusterRoleBinding is not reverted by OpenShift.
kubectl patch clusterrolebinding.rbac self-provisioners -p '{ "metadata": { "annotations": { "rbac.authorization.kubernetes.io/autoupdate": "false" } } }'
This example extends the default Kubernetes admin role so tenant owners gain admin privileges on all namespaces within their tenant. The extension adds:
restricted-v2 and nonroot-v2)kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: extend-admin-role
labels:
rbac.authorization.k8s.io/aggregate-to-admin: 'true'
rules:
- verbs:
- update
apiGroups:
- capsule.clastix.io
resources:
- '*/finalizers'
- apiGroups:
- security.openshift.io
resources:
- securitycontextconstraints
resourceNames:
- restricted-v2
- nonroot-v2
verbs:
- 'use'
The jobs that Capsule uses can be run with the restricted-v2 SCC, so their securityContext and podSecurityContext must be disabled. For Capsule itself, they are left enabled because Capsule runs as nonroot-v2, which is still a very secure SCC. Always set pullPolicy: Always on a multi-tenant cluster to ensure the intended images are used.
The following chart values can be used:
podSecurityContext:
enabled: true
securityContext:
enabled: true
jobs:
podSecurityContext:
enabled: false
securityContext:
enabled: false
image:
pullPolicy: Always
manager:
image:
pullPolicy: Always
Deploy the Capsule Helm chart with (at least) these values.
If etcd encryption is enabled on your OpenShift cluster and you are using TenantResource or GlobalTenantResource to replicate ConfigMaps or Secrets, the Capsule replication webhook must be configured to skip requests from OpenShift’s storage version migrator. Without this exclusion, the migrator, which iterates over all Secrets and ConfigMaps to rotate the encryption key, will be blocked or cause reconciliation errors, because the webhook intercepts its requests.
Add the following matchCondition to the replication webhook via the Helm chart values:
webhooks:
hooks:
replications:
matchConditions:
- name: "exclude-privileged-users"
expression: >
!(
request.userInfo.username in [
"system:serviceaccount:openshift-kube-storage-version-migrator:kube-storage-version-migrator-sa"
]
)
A minimal example tenant looks like the following:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
imagePullPolicies:
- Always
permissions:
matchOwners:
- matchLabels:
team: devops
priorityClasses:
allowed:
- openshift-user-critical
rules:
- audience:
- kind: Custom
name: "CapsuleUser"
enforce:
action: deny
metadata:
- apiGroups:
- "v1"
kinds:
- "Namespace"
labels:
".*openshift.io/.*":
required: false
"pod-security.kubernetes.io/enforce":
required: false
annotations:
".*openshift.io/.*":
required: false
"pod-security.kubernetes.io/enforce":
required: false
Combined with a TenantOwner resource to grant access to the tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
labels:
team: devops
name: devops
spec:
kind: Group
name: "oidc:org:devops:a"
More information about tenants and tenant owners can be found in the chapter Tenants.
For Capsule Proxy, all (pod)SecurityContexts can be disabled. By disabling these, the proxy and its jobs run under the nonroot-v2 SCC.
This example also enables the ProxyAllNamespaced feature, which is one of the Proxy’s most powerful capabilities.
The following helm values can be used as a template:
global:
jobs:
kubectl:
securityContext:
enabled: false
securityContext:
enabled: false
podSecurityContext:
enabled: false
options:
generateCertificates: false #set to false, since we are using cert-manager in .Values.certManager.generateCertificates
enableSSL: true
extraArgs:
- '--feature-gates=ProxyAllNamespaced=true'
image:
pullPolicy: Always
webhooks:
enabled: true
certManager:
generateCertificates: true
ingress:
enabled: true
annotations:
route.openshift.io/termination: "reencrypt"
route.openshift.io/destination-ca-certificate-secret: capsule-proxy-root-secret
hosts:
- host: "capsule-proxy.example.com"
paths: ["/"]
That is all the configuration needed for Capsule Proxy.
If the ProxyClusterScoped feature gate is enabled (not used in the example above, but is is a option to enable), it is recommended to create a GlobalProxySetting for Projects. OpenShift Projects are the equivalent of Kubernetes Namespaces, and when ProxyClusterScoped is active they can no longer be listed by default.
apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
name: openshift-projects
spec:
rules:
- subjects:
- kind: Group
name: oidc:org:devops:a
clusterResources:
- apiGroups:
- "project.openshift.io"
resources:
- "projects"
operations:
- List
selector:
matchLabels:
capsule.clastix.io/tenant: solar
On hosted control planes each hosted cluster runs its own OAuth server, exposed on its own route (for example, oauth-clusters-<cluster-name>.<base-domain>) with a certificate signed by that cluster’s root CA.
The oc login --web flow retrieves the OAuth server metadata from /.well-known/oauth-authorization-server on the API endpoint before any credentials exist. Since Capsule Proxy requires authentication on every request, this lookup is rejected by default and the login aborts with unsupported protocol scheme "".
Public paths solve this (available since capsule-proxy v0.14.1): requests to a public path are proxied to the upstream API server without authentication, while everything else keeps requiring one.
options:
extraArgs:
- '--public-paths=/.well-known/oauth-authorization-server'
With this in place oc login <capsule-proxy-host> --web completes: the browser login and the token exchange run against the hosted cluster’s OAuth server, and the API traffic goes through the proxy as usual.
Since the browser and the token exchange talk to the OAuth server directly, the client must trust the certificate of that OAuth server: import the hosted cluster’s root CA on the client first, the same procedure as when logging in against the API server directly.
The OpenShift console can be customized. For example, the capsule-proxy can be added as a shortcut on the top right application menu with the ConsoleLink CR:
apiVersion: console.openshift.io/v1
kind: ConsoleLink
metadata:
name: capsule-proxy-consolelink
spec:
applicationMenu:
imageURL: 'https://github.com/projectcapsule/capsule/raw/main/assets/logo/capsule.svg'
section: 'Capsule'
href: 'https://capsule-proxy.example.com'
location: ApplicationMenu
text: 'Capsule Proxy Kubernetes API'
It’s also possible to add links specific for certain namespaces, which are shown on the Namespace/Project overview. These can also be tenant specific by adding a NamespaceSelector:
apiVersion: console.openshift.io/v1
kind: ConsoleLink
metadata:
name: namespaced-consolelink-solar
spec:
text: "solar Docs"
href: "https://linktothesolardocs.com"
location: "NamespaceDashboard"
namespaceDashboard:
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: In
values:
- solar
Also a custom logo can be provided, for example by adding the Capsule logo.
Add these config lines to the existing cluster CR Console.
kubectl create configmap console-capsule-logo --from-file capsule-logo.png -n openshift-config
apiVersion: operator.openshift.io/v1
kind: Console
metadata:
name: cluster
spec:
customization:
customLogoFile:
key: capsule-logo.png
name: console-capsule-logo
customProductName: Capsule OpenShift Cluster
You now have a fully configured Capsule and Capsule Proxy installation on OpenShift, including console customizations, and the environment is ready to hand off to development teams.
The integration between Rancher and Capsule, aims to provide a multi-tenant Kubernetes service to users, enabling:
to end-users.
Tenant users will have the ability to access Kubernetes resources through:
On the other side, administrators need to manage the Kubernetes clusters through Rancher.
Rancher provides a feature called Projects to segregate resources inside a common domain. At the same time Projects doesn’t provide way to segregate Kubernetes cluster-scope resources.
Capsule as a project born for creating a framework for multi-tenant platforms, integrates with Rancher Projects enhancing the experience with Tenants.
Capsule allows tenants isolation and resources control in a declarative way, while enabling a self-service experience to tenants. With Capsule Proxy users can also access cluster-wide resources, as configured by administrators at Tenant custom resource-level.
You can read in detail how the integration works and how to configure it, in the following guides.
This guide explains how to setup the integration between Capsule and Rancher Projects.
It then explains how for the tenant user, the access to Kubernetes resources is transparent.
Cluster Role in RancherYou can follow this general guide to configure an OIDC authentication for Kubernetes.
For a Keycloak specific setup you can check this resources list.
/auth makes Rancher crashA custom Rancher Cluster Role is needed to allow Tenant users, to read cluster-scope resources and Rancher doesn’t provide e built-in Cluster Role with this tailored set of privileges.
When logged-in to the Rancher UI as administrator, from the Users & Authentication page, create a Cluster Role named Tenant Member with the following privileges:
get, list, watch operations over IngressClasses resources.get, list, watch operations over StorageClasses resources.get, list, watch operations over PriorityClasses resources.get, list, watch operations over Nodes resources.get, list, watch operations over RuntimeClasses resources.When onboarding tenants, the administrator needs to create the following, in order to bind the Project with the Tenant:
In Rancher, create a Project.
In the target Kubernetes cluster, create a Tenant, with the following specification:
kind: Tenant
...
spec:
namespaceOptions:
additionalMetadata:
annotations:
field.cattle.io/projectId: ${CLUSTER_ID}:${PROJECT_ID}
labels:
field.cattle.io/projectId: ${PROJECT_ID}
where $CLUSTER_ID and $PROEJCT_ID can be retrieved, assuming a valid $CLUSTER_NAME, as:
CLUSTER_NAME=foo
CLUSTER_ID=$(kubectl get cluster -n fleet-default ${CLUSTER_NAME} -o jsonpath='{.status.clusterName}')
PROJECT_IDS=$(kubectl get projects -n $CLUSTER_ID -o jsonpath="{.items[*].metadata.name}")
for project_id in $PROJECT_IDS; do echo "${project_id}"; done
More on declarative Projects here.
In the identity provider, create a user with correct OIDC claim of the Tenant.
In Rancher, add the new user to the Project with the Read-only Role.
In Rancher, add the new user to the Cluster with the Tenant Member Cluster Role.
A custom Project Role is needed to allow Tenant users, with minimum set of privileges and create and delete Namespaces.
Create a Project Role named Tenant Member that inherits the privileges from the following Roles:
When the configuration administrative tasks have been completed, the tenant users are ready to use the Kubernetes cluster transparently.
For example can create Namespaces in a self-service mode, that would be otherwise impossible with the sole use of Rancher Projects.
From the tenant user perspective both CLI and the UI are valid interfaces to communicate with.
kubectl-logs in to the OIDC providerthe Namespace is now part of both the Tenant and the Project.
As administrator, you can verify with:
kubectl get tenant ${TENANT_NAME} -o jsonpath='{.status}' kubectl get namespace -l field.cattle.io/projectId=${PROJECT_ID}
the Namespace is now part of both the Tenant and the Project.
As administrator, you can verify with:
kubectl get tenant ${TENANT_NAME} -o jsonpath='{.status}' kubectl get namespace -l field.cattle.io/projectId=${PROJECT_ID}
Before proceeding is recommended to read the official Rancher documentation about Project Monitors.
In summary, the setup is composed by a cluster-level Prometheus, Prometheus Federator via which single Project-level Prometheus federate to.
Before proceeding is recommended to read the official Capsule documentation about NetworkPolicy at Tenant-level`.
As Rancher’s Project Monitor deploys the Prometheus stack in a Namespace that is not part of neither the Project nor the Tenant Namespaces, is important to apply the label selectors in the NetworkPolicy ingress rules to the Namespace created by Project Monitor.
That Project monitoring Namespace will be named as cattle-project-<PROJECT_ID>-monitoring.
For example, if the NetworkPolicy is configured to allow all ingress traffic from Namespace with label capsule.clastix.io/tenant=foo, this label is to be applied to the Project monitoring Namespace too.
Then, a NetworkPolicy can be applied at Tenant-level with Capsule GlobalTenantResources. For example it can be applied a minimal policy for the wind Tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: wind-networkpolicies
spec:
tenantSelector:
matchLabels:
capsule.clastix.io/tenant: wind
resyncPeriod: 360s
pruningOnDelete: true
resources:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: wind
rawItems:
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: wind-minimal
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
# Intra-Tenant
- from:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: wind
# Rancher Project Monitor stack
- from:
- namespaceSelector:
matchLabels:
role: monitoring
# Kubernetes nodes
- from:
- ipBlock:
cidr: 192.168.1.0/24
egress:
# Kubernetes DNS server
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
# Intra-Tenant
- to:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: wind
# Kubernetes API server
- to:
- ipBlock:
cidr: 10.43.0.1/32
ports:
- port: 443
This guide explains how to setup the integration between Capsule Proxy and Rancher Projects.
It then explains how for the tenant user, the access to Kubernetes cluster-wide resources is transparent.
In order to integrate the Rancher Shell with Capsule it’s needed to route the Kubernetes API requests made from the shell, via Capsule Proxy.
The capsule-rancher-addon allows the integration transparently.
Add the Clastix Helm repository https://clastix.github.io/charts.
By updating the cache with Clastix’s Helm repository a Helm chart named capsule-rancher-addon is available.
Install keeping attention to the following Helm values:
proxy.caSecretKey: the Secret key that contains the CA certificate used to sign the Capsule Proxy TLS certificate (it should be"ca.crt" when Capsule Proxy has been configured with certificates generated with Cert Manager).proxy.servicePort: the port configured for the Capsule Proxy Kubernetes Service (443 in this setup).proxy.serviceURL: the name of the Capsule Proxy Service (by default "capsule-proxy.capsule-system.svc" hen installed in the capsule-system Namespace).In both CLI and dashboard use cases, the Cluster Agent is responsible for the two-way communication between Rancher and the downstream cluster.
In a standard setup, the Cluster Agents communicates to the API server. In this setup it will communicate with Capsule Proxy to ensure filtering of cluster-scope resources, for Tenants.
Cluster Agents accepts as arguments:
KUBERNETES_SERVICE_HOST environment variableKUBERNETES_SERVICE_PORT environment variablewhich will be set, at cluster import-time, to the values of the Capsule Proxy Service. For example:
KUBERNETES_SERVICE_HOST=capsule-proxy.capsule-system.svcKUBERNETES_SERVICE_PORT=9001. You can skip it by installing Capsule Proxy with Helm value service.port=443.The expected CA is the one for which the certificate is inside the kube-root-ca ConfigMap in the same Namespace of the Cluster Agent (cattle-system).
Capsule Proxy needs to provide a x509 certificate for which the root CA is trusted by the Cluster Agent. The goal can be achieved by, either using the Kubernetes CA to sign its certificate, or by using a dedicated root CA.
Note: this can be achieved when the Kubernetes root CA keypair is accessible. For example is likely to be possibile with on-premise setup, but not with managed Kubernetes services.
With this approach Cert Manager will sign certificates with the Kubernetes root CA for which it’s needed to be provided a Secret.
kubectl create secret tls -n capsule-system kubernetes-ca-key-pair --cert=/path/to/ca.crt --key=/path/to/ca.key
When installing Capsule Proxy with Helm chart, it’s needed to specify to generate Capsule Proxy Certificates with Cert Manager with an external ClusterIssuer:
certManager.externalCA.enabled=truecertManager.externalCA.secretName=kubernetes-ca-key-paircertManager.generateCertificates=trueand disable the job for generating the certificates without Cert Manager:
options.generateCertificates=falseIn order to allow tenant users to list cluster-scope resources, like Nodes, Tenants need to be configured with proper proxySettings, for example:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: wind
spec:
owners:
- kind: User
name: alice
proxySettings:
- kind: Nodes
operations:
- List
[...]
Also, in order to assign or filter nodes per Tenant, it’s needed labels on node in order to be selected:
kubectl label node worker-01 capsule.clastix.io/tenant=wind
and a node selector at Tenant level:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: wind
spec:
nodeSelector:
capsule.clastix.io/tenant: wind
[...]
The final manifest is:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: wind
spec:
owners:
- kind: User
name: alice
proxySettings:
- kind: Node
operations:
- List
nodeSelector:
capsule.clastix.io/tenant: wind
The same appplies for:
NodesStorageClassesIngressClassesPriorityClassesMore on this in the official documentation.
These instructions is specific to a setup made with Keycloak as an OIDC identity provider.
groupsclient audiencefull_group_pathMore on this on the official guide.
Configure an OIDC authentication provider, with Client with issuer, return URLs specific to the Keycloak setup.
Use old and Rancher-standard paths with
/authsubpath (see issues below).Add custom paths, remove
/authsubpath in return and issuer URLs.
CapsuleConfiguration by adding the "keycloakoidc_group://capsule.clastix.io" Kubernetes Group.get of Cluster.Tenant.proxySettings for the Tenant to enable tenant users to access cluster-wide resources.Capsule Operator can be easily installed on a Managed Kubernetes Service. Since you do not have access to the Kubernetes APIs Server, you should check with the provider of the service:
the default cluster-admin ClusterRole is accessible the following Admission Webhooks are enabled on the APIs Server:
PodNodeSelectorLimitRangerResourceQuotaMutatingAdmissionWebhookValidatingAdmissionWebhookThis is an example of how to install AWS EKS cluster and one user manged by Capsule. It is based on Using IAM Groups to manage Kubernetes access
Create EKS cluster:
export AWS_DEFAULT_REGION="eu-west-1"
export AWS_ACCESS_KEY_ID="xxxxx"
export AWS_SECRET_ACCESS_KEY="xxxxx"
eksctl create cluster \
--name=test-k8s \
--managed \
--node-type=t3.small \
--node-volume-size=20 \
--kubeconfig=kubeconfig.conf
Create AWS User alice using CloudFormation, create AWS access files and kubeconfig for such user:
cat > cf.yml << EOF
Parameters:
ClusterName:
Type: String
Resources:
UserAlice:
Type: AWS::IAM::User
Properties:
UserName: !Sub "alice-${ClusterName}"
Policies:
- PolicyName: !Sub "alice-${ClusterName}-policy"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: AllowAssumeOrganizationAccountRole
Effect: Allow
Action: sts:AssumeRole
Resource: !GetAtt RoleAlice.Arn
AccessKeyAlice:
Type: AWS::IAM::AccessKey
Properties:
UserName: !Ref UserAlice
RoleAlice:
Type: AWS::IAM::Role
Properties:
Description: !Sub "IAM role for the alice-${ClusterName} user"
RoleName: !Sub "alice-${ClusterName}"
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
Action: sts:AssumeRole
Outputs:
RoleAliceArn:
Description: The ARN of the Alice IAM Role
Value: !GetAtt RoleAlice.Arn
Export:
Name:
Fn::Sub: "${AWS::StackName}-RoleAliceArn"
AccessKeyAlice:
Description: The AccessKey for Alice user
Value: !Ref AccessKeyAlice
Export:
Name:
Fn::Sub: "${AWS::StackName}-AccessKeyAlice"
SecretAccessKeyAlice:
Description: The SecretAccessKey for Alice user
Value: !GetAtt AccessKeyAlice.SecretAccessKey
Export:
Name:
Fn::Sub: "${AWS::StackName}-SecretAccessKeyAlice"
EOF
eval aws cloudformation deploy --capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides "ClusterName=test-k8s" \
--stack-name "test-k8s-users" --template-file cf.yml
AWS_CLOUDFORMATION_DETAILS=$(aws cloudformation describe-stacks --stack-name "test-k8s-users")
ALICE_ROLE_ARN=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"RoleAliceArn\") .OutputValue")
ALICE_USER_ACCESSKEY=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"AccessKeyAlice\") .OutputValue")
ALICE_USER_SECRETACCESSKEY=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"SecretAccessKeyAlice\") .OutputValue")
eksctl create iamidentitymapping --cluster="test-k8s" --arn="${ALICE_ROLE_ARN}" --username alice --group capsule.clastix.io
cat > aws_config << EOF
[profile alice]
role_arn=${ALICE_ROLE_ARN}
source_profile=alice
EOF
cat > aws_credentials << EOF
[alice]
aws_access_key_id=${ALICE_USER_ACCESSKEY}
aws_secret_access_key=${ALICE_USER_SECRETACCESSKEY}
EOF
eksctl utils write-kubeconfig --cluster=test-k8s --kubeconfig="kubeconfig-alice.conf"
cat >> kubeconfig-alice.conf << EOF
- name: AWS_PROFILE
value: alice
- name: AWS_CONFIG_FILE
value: aws_config
- name: AWS_SHARED_CREDENTIALS_FILE
value: aws_credentials
EOF
Export “admin” kubeconfig to be able to install Capsule:
export KUBECONFIG=kubeconfig.conf
Install Capsule and create a tenant where alice has ownership. Use the default Tenant example:
kubectl apply -f https://raw.githubusercontent.com/clastix/capsule/master/config/samples/capsule_v1beta1_tenant.yaml
Based on the tenant configuration above the user alice should be able to create namespace. Switch to a new terminal and try to create a namespace as user alice:
# Unset AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY if defined
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
kubectl create namespace test --kubeconfig="kubeconfig-alice.conf"
This reference implementation introduces the recommended starting (baseline) infrastructure architecture for implementing a multi-tenancy Azure AKS cluster using Capsule. See CoAKS.
Canonical Charmed Kubernetes is a Kubernetes distribution coming with out-of-the-box tools that support deployments and operational management and make microservice development easier. Combined with Capsule, Charmed Kubernetes allows users to further reduce the operational overhead of Kubernetes setup and management.
The Charm package for Capsule is available to Charmed Kubernetes users via Charmhub.io.
This is general advice you should consider before making Kubernetes Distribution consideration. They are partly relevant for Multi-Tenancy with Capsule.
For a complete configuration combining the workload and networking policies, see the combined Tenant baseline.
User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider). The rationale is that other central platform components, such as ArgoCD, Grafana, Headlamp, or Harbor, should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions.
Capsule relies on native Kubernetes RBAC, so it’s important to consider how the Kubernetes API handles user authentication.
By default, Kubernetes clusters pull images directly from upstream registries like docker.io, quay.io, ghcr.io, or gcr.io. In production environments, this can lead to issues, especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled).
To ensure availability, performance, and control over container images, it’s essential to provide an on-premise OCI mirror.
This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., docker.io).
This way, all nodes automatically benefit from caching without requiring developers to change image URLs.
In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical. Instead, a centralized secrets management system should be established, such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project OpenBao (formerly the Vault community fork).
To integrate these external secret stores with Kubernetes, the External Secrets Operator (ESO) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing.
If no external secret store is available, there should at least be a secure way to store sensitive data in Git. In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case; called the sops-operator.
Use this complete Tenant as a starting point for application namespaces. It
combines the compatible baseline policies from Workloads
and Networking into one ordered
spec.rules list. All namespaces in this Tenant receive the same baseline.
| Area | Combined configuration |
|---|---|
| Resources | Default CPU, memory, and ephemeral-storage requests; remove CPU limits; match memory limits to requests; default ephemeral-storage limits. |
| Capacity | Share one illustrative quota across all Tenant namespaces. |
| QoS | Deny BestEffort after resource mutation. |
| Placement | Use the default scheduler, exclude control-plane nodes, deny control-plane and wildcard tolerations, and reject Pods created with nodeName. |
| Workload types | Deny DaemonSets. |
| Pod isolation | Set hostUsers: false and manage the namespace’s Pod Security labels at restricted. |
| Images | Allow the approved registry path and require Always pull policy. |
| Priority | Default to and allow only tenant-standard. |
| Services | Allow only ClusterIP and deny every external IP. |
| Namespace trust | Deny adding or changing company.com/system on Tenant namespaces. |
solar, solar-owner, and registry.example.com/solar with your
Tenant, owner, and approved registry path. Publish immutable image references
in that registry; the example’s registry matcher does not require a digest.tenant-standard PriorityClass
before admitting workloads that use this Tenant. The Tenant references the
class; it does not create it.hostUsers: false.
For incompatible workloads, define a separate, reviewed namespace profile.v1.36. Applications must supply security
contexts that satisfy restricted; these rules do not generate those fields.Explore the ordered rules below. Select a card to open its documentation, or view, copy, and download the complete Tenant YAML, including ownership and PriorityClass settings.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
# Default and allow the tenant-standard PriorityClass.
priorityClasses:
default: tenant-standard
allowed: [tenant-standard]
rules:
# One shared budget across all namespaces in this Tenant.
- quota:
- name: application-budget
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.memory: 40Gi
requests.ephemeral-storage: 100Gi
limits.ephemeral-storage: 200Gi
pods: "100"
services: "100"
# Apply placement and user-namespace settings on Pod creation.
- mutate:
- action: merge
workloads:
security:
hostUsers: false
placement:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
- key: node-role.kubernetes.io/master
operator: DoesNotExist
enforce:
action: allow
workloads:
targets: [pod]
placement:
schedulers:
- exact: [default-scheduler]
# Defaults for application containers, sidecars, and init containers.
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
ephemeral-storage:
policy: Default
value: 1Gi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
ephemeral-storage:
policy: Default
value: 2Gi
# Normalize limits when a workload supplies Pod-level resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Deny BestEffort Pods after resource mutation.
- enforce:
action: deny
workloads:
targets: [pod]
qosClasses: [BestEffort]
# Reject control-plane and wildcard tolerations.
- enforce:
action: deny
workloads:
targets: [pod]
placement:
tolerations:
- key:
exact:
- node-role.kubernetes.io/control-plane
- node-role.kubernetes.io/master
- key: {exp: '^$'}
operators: [Exists]
# Check direct assignment only at creation; allow scheduler-bound updates.
- enforce:
action: deny
conditions:
- name: direct-node-assignment
expression: >-
request.operation == 'CREATE' && has(object.spec.nodeName) && object.spec.nodeName != ''
workloads:
targets: [pod]
# Keep kind-level denials separate from workload property checks.
- enforce:
action: deny
workloads:
targets: [daemonset]
# Allow approved images with Always pull policy.
- enforce:
action: allow
workloads:
targets:
- pod/containers
- pod/initcontainers
- pod/ephemeralcontainers
- pod/volumes
registries:
- exp: '^registry\.example\.com/solar/.+$'
policy: [Always]
# Allow only ClusterIP Services.
- enforce:
action: allow
services:
types: [ClusterIP]
# Deny all Service external IPs.
- enforce:
action: deny
services:
externalIPs: {}
# Apply and reconcile Pod Security labels on the Namespace itself.
- enforce:
action: allow
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
pod-security.kubernetes.io/enforce:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/enforce-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/warn:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/warn-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/audit:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/audit-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
# Reserve this trusted-platform label for administrator-managed namespaces.
- enforce:
action: deny
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
company.com/system:
values:
- exp: ".*"
One shared budget across all namespaces in this Tenant.
- quota:
- name: application-budget
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.memory: 40Gi
requests.ephemeral-storage: 100Gi
limits.ephemeral-storage: 200Gi
pods: "100"
services: "100"
Apply placement and user-namespace settings on Pod creation.
- mutate:
- action: merge
workloads:
security:
hostUsers: false
placement:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
- key: node-role.kubernetes.io/master
operator: DoesNotExist
enforce:
action: allow
workloads:
targets: [pod]
placement:
schedulers:
- exact: [default-scheduler]
Defaults for application containers, sidecars, and init containers.
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
ephemeral-storage:
policy: Default
value: 1Gi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
ephemeral-storage:
policy: Default
value: 2Gi
Normalize limits when a workload supplies Pod-level resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
Deny BestEffort Pods after resource mutation.
- enforce:
action: deny
workloads:
targets: [pod]
qosClasses: [BestEffort]
Reject control-plane and wildcard tolerations.
- enforce:
action: deny
workloads:
targets: [pod]
placement:
tolerations:
- key:
exact:
- node-role.kubernetes.io/control-plane
- node-role.kubernetes.io/master
- key: {exp: '^$'}
operators: [Exists]
Check direct assignment only at creation; allow scheduler-bound updates.
- enforce:
action: deny
conditions:
- name: direct-node-assignment
expression: >-
request.operation == 'CREATE' && has(object.spec.nodeName) && object.spec.nodeName != ''
workloads:
targets: [pod]
Keep kind-level denials separate from workload property checks.
- enforce:
action: deny
workloads:
targets: [daemonset]
Allow approved images with Always pull policy.
- enforce:
action: allow
workloads:
targets:
- pod/containers
- pod/initcontainers
- pod/ephemeralcontainers
- pod/volumes
registries:
- exp: '^registry\.example\.com/solar/.+$'
policy: [Always]
Allow only ClusterIP Services.
- enforce:
action: allow
services:
types: [ClusterIP]
Deny all Service external IPs.
- enforce:
action: deny
services:
externalIPs: {}
Apply and reconcile Pod Security labels on the Namespace itself.
- enforce:
action: allow
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
pod-security.kubernetes.io/enforce:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/enforce-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/warn:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/warn-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/audit:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/audit-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
Reserve this trusted-platform label for administrator-managed namespaces.
- enforce:
action: deny
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
company.com/system:
values:
- exp: ".*"
Capsule applies mutations before validation. On Pod creation, the resource
rules supply missing requests and storage limits, remove CPU limits, and align
memory limits with requests. Explicit requests and storage limits survive the
Default policies. Placement mutation adds the control-plane exclusions and
sets hostUsers: false; enforcement then checks the resulting Pod.
The workload policies target Pods, including controller-created Pods. The DaemonSet rule separately rejects the controller itself. Add explicit controller targets if you also want resource or image checks on saved controller templates. Keep the DaemonSet denial and conditional direct-assignment denial separate: adding workload properties to a targets-only rule changes its meaning.
The quota rule generates solar-application-budget, a
GlobalResourceQuota shared
across this Tenant’s namespaces. It bounds aggregate consumption; the resource
defaults do not impose maximum requests or reserve node capacity by themselves.
Retain capacity for failover and system services as described in
capacity planning.
The restricted Pod Security labels are managed values. The platform-label
denial checks added or changed metadata; unchanged existing values are skipped
on updates. Remove any existing tenant-controlled company.com/system labels
before using them as a network trust boundary.
The memory-overcommit and ExternalName examples on the individual pages are deliberate exceptions. Add them only under reviewed namespace selectors, with the selecting labels under platform control. Review rule order and scope when composing further rules so a later decision does not unintentionally relax the baseline.
Apply the native NetworkPolicy GlobalTenantResource alongside this Tenant. It distributes default-deny, same-Tenant, DNS, and platform-ingress policies; NetworkPolicies are not embedded in this Tenant manifest. Review its CNI requirements and the permissions needed to prevent tenants from adding policies that widen access.
Use the additional admission policies
for requirements such as PDB validation, emptyDir.sizeLimit, debugging access,
and certificate issuer restrictions. Those examples require their respective
admission mechanisms and are not installed by this Tenant.
Configure authentication, registry mirrors, and secret management at platform level. Review existing workloads and Services before rollout; admission policies do not remove previously admitted resources.
Use namespace rules to establish workload defaults and validate the resulting
Pods. Every Capsule policy example below is a complete Tenant manifest with an
owner and spec.rules. Replace solar and solar-owner with your Tenant name
and owner identity. Each example is an independent configuration; to adopt
multiple policies, combine their spec.rules entries into one Tenant manifest.
Namespace selectors can give different namespaces different profiles; keep the
labels selecting mandatory profiles under platform control.
See the combined Tenant baseline for resource, placement, image, and networking policies together. Add the PDB policy below for namespace profiles that must support routine maintenance.
Resource settings are both a scheduling contract and a runtime safety boundary. Kubernetes schedules a Pod from its resource requests, while the kubelet and container runtime enforce its resource limits. Correct values help the scheduler place Pods safely, keep one workload from affecting its neighbours, and give autoscalers meaningful data.
CPU and memory need different treatment:
See Resource Management for Pods and Containers for the complete Kubernetes behavior.
Teams commonly use one of the following patterns:
BestEffort QoS. The scheduler
reserves no CPU or memory, and any memory use exceeds its memory request.
This is suitable only for disposable workloads.Burstable QoS. The scheduler accounts for the
workload and it can use spare capacity. Without a memory limit, however, a
leak can consume all memory available on the node.Burstable QoS and allows
controlled bursts. CPU can be throttled at its limit, and memory above the
request is more exposed during node pressure.Guaranteed QoS. It
reserves the full declared capacity and reduces memory-pressure exposure,
but prevents CPU bursting and can lower cluster utilization.Burstable QoS. It preserves CPU bursting while
reserving and bounding memory. This is the recommended baseline for general
workloads.A CPU limit can be useful when a workload needs a fixed CPU ceiling. Exclusive
CPU allocation under the CPU Manager static
policy
additionally requires eligible Guaranteed containers with integer CPU
requests. Guaranteed should not be the automatic target for every production
Pod.
For every application container, sidecar, and init container:
For example:
apiVersion: v1
kind: Pod
metadata:
name: api
spec:
containers:
- name: api
image: registry.example.com/solar/api:1.0.0
imagePullPolicy: Always
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
memory: 512Mi
The baseline can be visualized as two independent resource decisions that produce one Pod QoS class:
flowchart TB workload["Container resources"] cpu["CPU<br/>request: 250m<br/>limit: none"] memory["Memory<br/>request: 512Mi<br/>limit: 512Mi"] cpuResult["Scheduled CPU share<br/>Can use spare CPU"] memoryResult["512Mi scheduled<br/>Bounded at 512Mi"] qos["Pod QoS: Burstable"] workload --> cpu workload --> memory cpu --> cpuResult memory --> memoryResult cpuResult --> qos memoryResult --> qos
This gives the scheduler an honest view of the CPU needed under contention and
the maximum memory the container can consume. It also lets the container use
idle CPU above 250m without being throttled by an artificial ceiling.
Declare the memory request explicitly. If a limit is present without a request and no admission mechanism supplies a default, Kubernetes copies the limit to the request. Relying on that default makes the intended scheduling contract less obvious to readers and policy tools.
Treat a different memory request and limit as a deliberate overcommit policy, not as a default. It can improve density, but a container using more memory than its request is more vulnerable to eviction when its node is under memory pressure.
Apply this baseline with workload resource rules:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Normalize Pod-level limits too, when a workload supplies Pod resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
Replace solar-owner with your owner identity and size the defaults for your
workloads. Default preserves existing requests, Remove deletes CPU limits,
and MatchRequest sets memory limits to the corresponding requests. These
resource policies mutate Pod creation before enforcement; ordinary Pod updates
are validated without resource mutation. They also cover controller-created
Pods. Add explicit controller targets
if you want template validation and resource mutation when controllers are saved.
OOMKilled container needs investigation and usually a
corrected value; increasing the limit blindly can hide a memory leak.Kubernetes derives a Pod’s Quality of Service (QoS) class from CPU and memory settings. With container-level resources:
Guaranteed: every container has equal, non-zero CPU and memory requests
and limits.Burstable: some CPU or memory resources are declared, but the Pod does not
meet the Guaranteed criteria.BestEffort: no CPU or memory requests or limits are declared.Where supported, Pod-level resources also participate in classification. Ephemeral-storage requests and limits do not determine QoS.
The recommended baseline intentionally produces Burstable Pods because CPU
limits are absent. Equal memory requests and limits make memory reservations
explicit, but a QoS class does not guarantee protection from eviction. See
Eviction and Capacity Planning for the
resource-specific decisions, including disk pressure.
Capsule can allow, deny, or audit QoS
classes. A good tenant baseline
is to reject BestEffort Pods and permit the intentional Burstable pattern
described above.
For example, reject BestEffort without forcing all workloads into
Guaranteed:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod]
qosClasses: [BestEffort]
Resource mutations run before this check. When combined with the recommended resource baseline, its defaults supply requests before QoS is evaluated. This QoS-only example does not add requests or limits, ensure every container is sized, or provide a disk or inode reservation.
QoS describes how resources are declared. A
PriorityClass
describes how important one Pod is relative to another. These mechanisms are
independent: a Burstable Pod can have a higher priority than a Guaranteed
Pod.
A PriorityClass is cluster-scoped and maps a name to an integer. Higher-priority Pods are considered earlier in the scheduling queue. By default, a pending higher-priority Pod can also preempt lower-priority Pods when that would make it schedulable. The kubelet also considers priority during node-pressure eviction. Priority does not reserve capacity and does not give a running container more CPU time; requests and limits still define that behavior.
Define a small set of classes with clear purposes. For example, a standard tenant class can be non-preempting:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: tenant-standard
value: 1000
preemptionPolicy: Never
globalDefault: false
description: "Default priority for tenant application workloads"
Workload owners select an approved class on the Pod template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
priorityClassName: tenant-standard
containers:
- name: api
image: registry.example.com/solar/api:1.0.0
imagePullPolicy: Always
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
memory: 512Mi
Use elevated, preempting classes only for workloads whose unavailability would
prevent the platform or other applications from recovering. Do not use them to
compensate for undersized requests or insufficient cluster capacity. In a
multi-tenant cluster, restrict which PriorityClasses each tenant may select;
otherwise a tenant could displace another tenant’s Pods. Capsule can allow
classes and assign a tenant
default, and Kubernetes
ResourceQuota can limit consumption by PriorityClass.
Application owners should size their workloads, while the platform supplies safe fallbacks and validation:
LimitRange for simple namespace defaults and minimum or maximum
values. Do not configure a default CPU limit, because that would reverse the
recommended policy.ResourceQuota, Global Resource
Quotas, or Resource
Pools to control aggregate tenant
consumption. Quotas are capacity boundaries; they do not replace accurate
per-container sizing.Before deploying a workload, verify that every container has a CPU request, no
CPU limit, and equal memory request and limit; that the expected QoS class is
Burstable; and that any selected PriorityClass is approved for the tenant.
The terms overprovisioning and overcommit are sometimes used interchangeably, although they describe different capacity strategies:
The right approach depends on the resource. CPU contention delays work; memory contention can terminate Pods. They should therefore have different defaults.
Kubernetes places Pods according to CPU requests, so the total CPU requests on a node normally cannot exceed its allocatable CPU. With no CPU limits, however, the potential CPU demand of the running workloads can be much higher than the node’s capacity. This is intentional CPU overcommit: containers can use otherwise idle CPU, and their requests determine their relative share when several containers need CPU at the same time.
The recommended baseline supports CPU overcommit safely when requests remain honest. Do not lower requests merely to fit more Pods onto a node. An undersized request makes scheduling and capacity reports misleading, reduces the workload’s CPU share during contention, and distorts CPU-utilization-based Horizontal Pod Autoscaling.
There is no universal safe CPU overcommit ratio. Choose it from workload measurements and service objectives, and account for correlated events such as traffic peaks, rollouts, batch schedules, and node failures. Watch for:
Memory is overcommitted when the sum of memory requests fits on a node but the workloads can collectively use more memory than the node provides. A common way to create this condition is to set memory requests lower than memory limits. The scheduler considers the lower requests, while each container can grow toward its higher limit.
Our baseline avoids deliberate application memory overcommit by setting each memory request equal to its limit. Node services, runtime overhead, and failover still need their own capacity budget; this setting alone does not prevent node memory pressure.
Use memory overcommit only as an explicit exception for workloads that are well understood, restart-tolerant, and unlikely to peak together. Keep a hard memory limit, retain node headroom, and expect increased eviction risk whenever usage exceeds the request. Avoid memory overcommit for critical or stateful workloads, uncertain memory profiles, and applications with expensive recovery.
Monitor the exception using both scheduling and runtime signals:
For a measured exception, this complete Tenant includes the recommended
baseline followed by a rule selecting only batch namespaces. The last rule
replaces the earlier MatchRequest limit policy for regular and init container
memory in namespaces labeled example.com/capacity-profile: batch:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Normalize Pod-level limits too, when a workload supplies Pod resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Override container memory limits only in batch namespaces.
- namespaceSelector:
matchLabels:
example.com/capacity-profile: batch
enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
limits:
memory:
policy: Ratio
value: "2"
A missing memory limit becomes twice its positive request; an explicit limit
above that ratio is denied. Earlier request defaults and CPU policies remain
in effect. If workloads also use Pod-level memory resources, configure that
limit policy for the intended burst too. A later Preserve policy instead
clears earlier management and constraints for that same target, field, and
resource; it does not undo
Kubernetes defaults. See resource policy ordering.
Even well-sized workloads need room for failover, rolling updates, sudden scale-outs, and the delay before new nodes become ready. Define the required headroom from those recovery objectives rather than relying on capacity that happens to be idle.
Headroom can be provided by keeping additional nodes running or, when using Cluster Autoscaler, by scheduling low-priority placeholder Pods with resource requests. Real workloads preempt the placeholders and use the reserved space immediately. The displaced placeholder Pods become pending and can trigger a replacement node. The Cluster Autoscaler overprovisioning guide describes this pattern.
Placeholder Pods need a dedicated priority below application Pods, but not below
the autoscaler’s expendable-Pod cutoff if they should trigger scale-up. The
application class must permit preemption to displace them: tenant-standard
with preemptionPolicy: Never above cannot do that. Test the replacement and
scale-down behavior, and keep application state out of placeholders. A
PriorityClass only decides which Pods give way; it does not create capacity
by itself.
Review reservation ratios separately for each node pool and tenant. Aggregate cluster averages can hide a saturated pool or a single noisy tenant. Revisit the ratios after workload growth, node-type changes, autoscaler changes, or a capacity-related incident.
Plan for the resource that can run out. A CPU/memory QoS class is neither a capacity reservation for other resources nor a general eviction ranking.
| Situation | What happens | What to plan |
|---|---|---|
| CPU contention | Work slows; CPU limits can throttle. CPU saturation is not a kubelet node-pressure eviction signal. | Realistic CPU requests, latency targets, and scale-out capacity. |
| Container memory limit or node OOM | The kernel can kill a container; its restart policy determines recovery. This differs from a kubelet evicting the Pod. | Measured memory bounds, system reservations, and investigation of OOMKilled. |
| Node memory pressure | Kubelet may evict Pods to reclaim memory. | Memory requests versus actual usage, priorities, and node headroom. |
| Disk space pressure | Kubelet first attempts reclamation and can then evict Pods. CPU/memory QoS is not protection. | Storage requests, logs, writable layers, image caches, and filesystem free space. |
| Inode or PID pressure | Pods can be evicted even with spare memory or disk bytes. Priority matters; these resources have no Pod requests. | File counts, process counts, and per-node limits. |
| Local ephemeral-storage limit exceeded | With storage accounting enabled, the Pod can be evicted even without node-wide disk pressure. | Per-container storage limits and bounded scratch volumes. |
For threshold handling and reclamation, see node-pressure eviction. CPU and memory limit behavior is described under resource management; local storage accounting has its own requirements and limits.
For a node-wide OOM, Linux process selection also uses kubelet-provided OOM score adjustments influenced by QoS and memory requests. That is a different mechanism from pressure-eviction ranking. See the kubelet OOM policy.
Scheduler preemption is a separate decision that makes room for a pending
higher-priority Pod; it can happen without node pressure. preemptionPolicy: Never prevents a Pod from preempting others, not from being a victim. See
priority and preemption.
A drain normally uses the Eviction API, which checks PodDisruptionBudgets. Kubelet node-pressure eviction does not honor a PDB; hard thresholds can terminate immediately. A PDB limits some voluntary disruptions and creates no spare capacity. See disruption budgets and the PDB maintenance policy below.
For memory, eviction ranking considers usage above the request first, then lower Pod priority, then usage minus the request. Disk-byte ranking uses storage consumption and requests for the affected filesystem, not CPU/memory QoS. Filesystem layout determines which usage is counted. Inode/PID pressure cannot compare against a Pod request for those resources. See the Kubernetes eviction ordering.
Consider two Pods at the same priority on a node using one filesystem for local Pod storage:
| Pod | CPU/memory QoS | Ephemeral-storage request | Local storage use |
|---|---|---|---|
| A | Guaranteed | None | 5Gi |
| B | Burstable | 8Gi | 4Gi |
Under disk-byte pressure, A exceeds its storage request while B does not. With
valid usage statistics, A ranks before B despite being Guaranteed. Adding CPU
limits to B would not fix the full disk. Monitor the actual filesystem signal;
inode exhaustion is a different case from running out of bytes.
Likewise, under memory pressure, a high-priority Pod using more than its memory request can rank before a lower-priority Pod still within its request. Pods within requests can still be evicted if the node cannot recover, for example when system daemons exceed their reservations.
Requests for ephemeral-storage inform scheduling. Limits cover accounted
container logs, writable layers, and disk-backed emptyDir usage, with
Pod-level accounting rules. They are not an instantaneous filesystem quota.
Memory-backed emptyDir consumes memory instead. Persistent volumes need their
own storage planning. See local ephemeral
storage and
memory-backed volumes.
For workloads with local scratch data, add measured storage defaults:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
ephemeral-storage:
policy: Default
value: 1Gi
limits:
ephemeral-storage:
policy: Default
value: 2Gi
These are fallback values, not maximum allowed sizes: Default preserves
explicit settings. Use quota to bound namespace consumption, configure log
rotation, and size image-cache space and inodes separately. Ephemeral-storage
policies cannot target Pod-level spec.resources.
24Gi allocatable memory each have only 48Gi after one fails, before
failover or rollout demand. 60Gi of requests would not fit that scenario.Evicted, OOMKilled, and FailedScheduling. kubectl top
covers CPU/memory usage; it does not establish disk, inode, or PID headroom.A PodDisruptionBudget (PDB) protects application availability during voluntary evictions. An overly restrictive budget can also stop a shared node from being emptied, delaying node upgrades or replacement. One tenant’s Pod can therefore hold up maintenance affecting other tenants. PDBs also constrain Cluster Autoscaler scale-down, so a blocked eviction can leave otherwise removable nodes running.
Common causes and the corresponding Capsule policies are:
| Cause | Operational effect | Capsule policy |
|---|---|---|
maxUnavailable: 0, minAvailable: "100%", or an integer minimum equal to the replica count | No healthy replica can be voluntarily evicted | Require evictableReplicas.min: 1 |
| Scaling down makes a percentage budget round up to all replicas | A previously usable budget permits no disruption; for example, minAvailable: "75%" allows one at four replicas but none at three | Check replica changes on the controller and /scale against the same minimum |
| Multiple PDBs select a Pod | The Eviction API can reject the request with HTTP 500 | Set allowOverlap: false |
An unhealthy Running Pod is covered by IfHealthyBudget while its application is already below budget | An unhealthy Pod can hold up a drain | Allow only AlwaysAllow in unhealthyPodEvictionPolicies |
The Eviction API
checks the live budget. AlwaysAllow permits unhealthy Running Pods to leave
even when that budget is unsatisfied; healthy Pods still obey it. An omitted
PDB eviction policy means IfHealthyBudget. See the
Kubernetes unhealthy eviction policies.
For a namespace profile that must support routine maintenance, use this complete Tenant example. It checks Pods and Deployment/StatefulSet templates, including PDB writes and supported replica changes:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- namespaceSelector:
matchLabels:
example.com/profile: production
enforce:
action: allow
workloads:
targets:
- pod
- deployment
- statefulset
disruptionBudgets:
allowOverlap: false
evictableReplicas:
min: 1
unhealthyPodEvictionPolicies:
- AlwaysAllow
The rule applies only to namespaces labeled example.com/profile: production.
Keep that label under platform control; omit namespaceSelector to apply the
policy throughout the Tenant. PDB authors must explicitly set
unhealthyPodEvictionPolicy: AlwaysAllow; Capsule validates it without adding
the field. No PDB is required or created. For the complete field and target
reference, see PDB enforcement.
Choose disruption allowances with the application owner. min: 1 permits the
only replica of a singleton to be evicted when its PDB allows that, so it cannot
promise uninterrupted service. Run enough replicas and place them across the
failure domains you need to tolerate. Add evictableReplicas.max only when you
also need an upper bound on the per-controller configured allowance. Scaling
to zero is exempt from the replica-count check; this policy does not impose a
minimum workload size.
status.disruptionsAllowed. After an eviction, a replacement may remain Pending
because of capacity, affinity, taints, or volume placement. Further evictions can
then be blocked despite a compliant policy. Reserve headroom
and test recovery on the nodes that will remain available.Before maintenance, inspect PDB currentHealthy, desiredHealthy, and
disruptionsAllowed alongside Pod readiness and scheduling events. Review all
matching selectors when an eviction fails. A policy change does not repair
existing objects, and concurrent admissions across separate resources can race.
Coordinate an application or budget repair with its owner when necessary;
deleting a conflicting PDB remains possible.
PDBs cover voluntary API evictions. Kubelet node-pressure eviction does not honor them, including under disk pressure. Keep resource sizing and eviction planning alongside these policies; a PDB does not provide spare resources or protect a node from exhaustion.
Use workload mutations to establish scheduling restrictions and enforcement to reject incompatible requests. The effective namespace profile is applied to new Pods, including Pods created by workload controllers.
Keep control-plane node role labels and taints managed by the platform. For ordinary tenant applications, add required node affinity that excludes both the current and legacy control-plane labels:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- mutate:
- action: merge
workloads:
placement:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
- key: node-role.kubernetes.io/master
operator: DoesNotExist
enforce:
action: allow
workloads:
targets: [pod]
placement:
schedulers:
- exact: [default-scheduler]
- enforce:
action: deny
workloads:
targets: [pod]
placement:
tolerations:
- key:
exact:
- node-role.kubernetes.io/control-plane
- node-role.kubernetes.io/master
# An empty key with Exists tolerates every taint key.
- key: {exp: '^$'}
operators: [Exists]
- enforce:
action: deny
conditions:
- name: direct-node-assignment
expression: >-
request.operation == 'CREATE' && has(object.spec.nodeName) && object.spec.nodeName != ''
workloads:
targets: [pod]
Affinity merge combines the exclusions with existing required constraints;
it does not replace an application’s other node restrictions. The scheduler
allow-list keeps this example on the native scheduler. The last rule rejects
Pods created with nodeName, which would bypass scheduling, while allowing
normal updates after the scheduler binds a Pod. Tolerations permit scheduling
onto tainted nodes; they do not select nodes or prevent kubelet pressure
evictions. See taints and
tolerations.
These settings apply to newly admitted Pods, not existing ones. Keep node
label/taint writes and pods/binding permissions restricted to platform
components. If you use another approved scheduler or different control-plane
labels, adapt the profile to that setup.
Reserve DaemonSets for platform-managed agents when application tenants should not run one Pod on every eligible node:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [daemonset]
With no workload properties, targets apply the action to the kind itself. This rejects DaemonSet creation and main-resource updates in the Tenant’s namespaces; existing objects are not removed. Other kinds are unaffected. Keep this rule separate from resource or image policies, because adding those properties changes targets into policy scopes. See workload targets.
For supported Linux node pools, mutate[].workloads.security.hostUsers: false
maps users inside a Pod to
different host IDs. Root inside the container therefore does not become host
root. Enable this only after verifying kernel, runtime, and volume filesystem
support; host namespaces and some volume configurations are incompatible. See
Kubernetes user namespaces.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- mutate:
- action: merge
workloads:
security:
hostUsers: false
This sets the value on Pod creation, even if the submitted value was true.
It does not change the container’s runAsUser or migrate existing Pods. Apply
it to namespaces whose workloads and nodes support user namespaces, and use
Pod Security Admission for the remaining security-context requirements.
Capsule does not configure node sysctls or runtime prerequisites.
Use Pod Security
Admission
to apply the privileged, baseline, or restricted standards. Start new
application profiles at restricted; use warning and audit modes to assess
existing workloads before enforcing a stricter profile.
Distribute the namespace labels through managed metadata
rules. Explicitly select Namespace;
wildcard kinds do not opt into Namespace metadata management. This example
pins the policy to v1.36; choose a policy version supported by your cluster
and review it during upgrades.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
pod-security.kubernetes.io/enforce:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/enforce-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/warn:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/warn-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/audit:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/audit-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
Capsule sets managed values during admission and reconciles them on existing
namespaces for this unconditional rule. A user’s conflicting label value is
overwritten. Kubernetes then checks admitted Pods against the selected
standard; changing namespace labels does not retroactively evict running Pods.
For different profiles, use namespace selectors whose profile labels are
controlled by the platform. Prefer narrow, reviewed exceptions over allowing
tenants to select privileged themselves.
Use an approved registry and immutable image references. When requiring
imagePullPolicy: Always, remember that the runtime can still reuse cached
layers; Always does not mean downloading every layer again. Private-image
credential verification also depends on the kubelet’s
KubeletEnsureSecretPulledImages support and verification policy. Review the
image pull and credential verification
behavior
for your Kubernetes version.
For a namespace profile that requires registry checks on each container start:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
workloads:
targets:
- pod/containers
- pod/initcontainers
- pod/ephemeralcontainers
- pod/volumes
registries:
- exp: '^registry\.example\.com/solar/.+$'
policy: [Always]
This permits only references in that registry path with Always pull policy;
it validates the submitted policy rather than changing it. Registry allow-listing
is separate from registry authentication and image signing. Budget image-cache
storage and registry availability as part of capacity planning.
For ordinary application namespaces, start with ClusterIP Services and expose
applications through a platform-managed Gateway or Ingress. Give namespaces
that need other Service types an explicit profile through
Service enforcement rules.
See the combined Tenant baseline for Service restrictions together with workload and namespace policies.
| Setting | Default policy for application namespaces | Reason |
|---|---|---|
ClusterIP | Allow, including headless Services. | Supports internal service discovery. NetworkPolicy still needs to control who can connect. |
NodePort | Deny. | Opens a port on node addresses, creating an exposure path outside the managed application entry point. |
LoadBalancer | Deny unless approved. | Can provision externally reachable infrastructure and normally allocates node ports too. |
ExternalName | Deny unless approved. | Lets a tenant alias a Service name to another DNS name; review destinations before enabling it. |
spec.externalIPs | Deny on every Service type. | Lets a Service claim traffic for an IP independently of its type. |
These are recommended rules to install, not Capsule’s automatic defaults. See the Kubernetes Service types for their networking behavior. The upstream external IP deprecation proposal explains the traffic-interception risk of user-selected external IPs.
This complete Tenant applies the baseline to all its namespaces:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
services:
types: [ClusterIP]
- enforce:
action: deny
services:
externalIPs: {}
Replace solar and solar-owner with your Tenant name and owner identity. The
type allow-list rejects NodePort, LoadBalancer, and ExternalName.
Under action: deny, externalIPs: {} rejects every supplied external IP,
including IPv4 and IPv6, while allowing Services without external IPs. This
separate check matters because a ClusterIP Service can also set
spec.externalIPs.
These checks run on Service creation and updates. They do not remove existing Services or close existing exposure when a rule is installed. Review existing Services during rollout; subsequent updates must satisfy the effective rules.
Keep the baseline for ordinary namespaces and add narrowly scoped exceptions for workloads that need them. Protect the namespace labels selecting an exception so tenant users cannot grant themselves a more permissive profile. See Namespace metadata rules.
For example, this complete Tenant retains the baseline and permits an
ExternalName pointing to database.example.com only in namespaces labeled
example.com/service-profile: external-database:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
services:
types: [ClusterIP]
- enforce:
action: deny
services:
externalIPs: {}
- namespaceSelector:
matchLabels:
example.com/service-profile: external-database
enforce:
action: allow
services:
types: [ExternalName]
externalNames:
hostnames:
- exact: [database.example.com]
In selected namespaces, ClusterIP remains allowed and the approved
ExternalName is added. Other hostnames, NodePort, LoadBalancer, and all
external IPs remain denied. Non-selected namespaces retain the baseline.
Hostname constraints alone do not allow the ExternalName type; the exception
must include both the type and its hostname constraint.
The examples are alternative configurations of the same Tenant. Combine the
desired spec.rules entries into one manifest when adopting other policies.
Review rule order and scope: later matching rules
can override earlier decisions for the same Service value.
An ExternalName allow-list checks the submitted DNS name, not its resolved IP
addresses. It does not restrict connections made directly to other names or
IPs. Keep egress controls in place and consider who controls the approved DNS
name. See ExternalName Services.
Some workloads need a dedicated load balancer. Treat that as a separate namespace profile, with a reviewed provider configuration:
LoadBalancer only in
selected namespaces. Require the appropriate class or provider annotations
for an internal load balancer where needed. Use
metadata rules for annotations and
enforcement conditions to deny
incompatible Service fields. An allowed address range alone does not make a
load balancer private.NodePort Service type does not stop
a permitted LoadBalancer from allocating node ports. If the implementation
routes directly to Pods, require spec.allocateLoadBalancerNodePorts: false
and reject explicit spec.ports[].nodePort values. Changing that flag on an
existing Service does not release previously allocated ports; remove them
explicitly. Other implementations need node ports, so constrain access with
node firewalls and provider controls. See load balancer node-port
allocation.spec.loadBalancerSourceRanges and verify the provider enforces them; an
unsupported provider can ignore this field. Capsule’s
LoadBalancer CIDR checks
validate both spec.loadBalancerIP and spec.loadBalancerSourceRanges.
They do not require source ranges specifically when loadBalancerIP is
present, so use a condition-based deny rule if missing source ranges must be
rejected. See the Service API.Service admission controls configuration; it does not establish tenant network isolation. Apply NetworkPolicies with a supporting CNI and test the actual ingress and egress paths. Address translation can affect which source or destination a policy sees, particularly with load balancers. See NetworkPolicy behavior.
Also review these paths outside the Service type rules:
EndpointSlice, legacy Endpoints, and
services/status through RBAC. Endpoint writes can redirect Service traffic;
keep them with trusted controllers unless a workload explicitly needs that
capability. See RBAC guidance on traffic
redirection.hostNetwork and hostPort. Blocking NodePort Services does not
restrict these Pod settings.ClusterIP backend can still be deliberately published by
those controllers; see Ingress enforcement.Use GlobalTenantResource to distribute a common network profile across Tenant namespaces. Start with default-deny ingress and egress, then allow the flows applications need. The CNI must support and enforce native Kubernetes NetworkPolicies.
Native NetworkPolicies are additive: an additional allow policy can widen the effective access. Protecting the replicated baseline alone is therefore not a mandatory isolation boundary. Restrict tenant permissions to create additional NetworkPolicies when the platform must own that boundary, or use an administrator-enforced policy mechanism supported by your CNI. Both source egress and destination ingress must allow a connection. See the Kubernetes policy-combination rules.
This example creates four policies per selected namespace. Verify replication
readiness and test same-Tenant access, cross-Tenant denial, DNS, and platform
ingress before relying on the profile. Replication readiness confirms resource
delivery; it does not prove the CNI has applied the rules. Tune
resyncPeriod for the number of namespaces you manage.
This complete, cluster-scoped GlobalTenantResource creates four native
networking.k8s.io/v1 NetworkPolicies in every namespace of every selected
Tenant. It has no metadata.namespace. tenantSelector: {} selects all
Tenants; replace it with a label selector to roll out the profile gradually.
Use resources[].namespaceSelector when only some namespaces need this profile.
The replication identity must
have permission to manage NetworkPolicies in the destination namespaces.
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: tenant-network-baseline
spec:
scope: Namespace
tenantSelector: {}
resyncPeriod: 60s
resources:
- policy:
creation: Owner
protect: true
deletion: Remove
rawItems:
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-default-deny
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-allow-same-tenant
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
ingress:
- from:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: "{{tenant.name}}"
egress:
- to:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: "{{tenant.name}}"
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-allow-dns
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
# Both selectors apply to the same destination Pod.
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-allow-platform-ingress
spec:
podSelector: {}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels:
company.com/system: "true"
Capsule renders {{tenant.name}} separately for each destination Tenant using
Fast Templates, and sets
the destination namespace on each generated policy. The selector uses Capsule’s
namespace membership label, capsule.clastix.io/tenant.
The resulting profile permits:
| Traffic | Allowance in this profile |
|---|---|
| Between Pods in the same Tenant, including the same namespace | Ingress and egress on all ports. |
| From Tenant Pods to cluster DNS | TCP and UDP port 53 to the selected DNS Pods. |
| From Pods in trusted platform namespaces to Tenant Pods | Ingress on all ports; narrow the Pod selectors and ports for your monitoring or ingress controllers. |
| To another Tenant, the internet, or other platform destinations | No allowance from this profile; add a reviewed policy for required flows. |
For namespace-level isolation within a Tenant, replace the same-tenant peers
with podSelector: {} so they select Pods only in the policy’s namespace.
Install the namespace metadata restriction below
on every application Tenant before trusting company.com/system.
Check your DNS deployment’s namespace and labels; the example assumes
kube-system and k8s-app: kube-dns. See DNS
troubleshooting.
NodeLocal DNSCache or host-networked DNS may need different CNI-specific
allowances. Test DNS over both UDP and TCP instead of relying on a fixed Service
IP. The platform workloads also need an egress policy permitting their
connections if their own namespaces isolate egress.
protect: true protects the replicated objects from tenant modification or
deletion through Capsule admission. It does not stop someone from creating
another NetworkPolicy. With deletion: Remove, removing the replication or a
destination from its selection removes the policies it created; plan those
changes as changes to network access.
The platform-ingress policy trusts namespaces carrying
company.com/system: "true". Reserve that label for administrator-managed
platform namespaces outside application Tenants. Restrict who can label those
namespaces through RBAC.
Use a metadata enforcement rule
to deny the label on application Tenant namespaces. This complete Tenant
replaces the legacy namespaceOptions.forbiddenLabels configuration:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
company.com/system:
values:
- exp: ".*"
The explicit Namespace kind targets the Namespace object itself. The rule
rejects the label on creation and when an update adds or changes its value,
including "true", "false", and an empty value. Namespaces without the label
and updates removing it remain allowed. Apply this rule to every application
Tenant and merge it with the Service rules above when using both profiles.
Keep it unconditional so changing a namespace label cannot opt out of the
restriction. Review later rules that could override this denial.
Metadata validation skips unchanged values on updates. This rule therefore
does not remove an existing label or reject an update that leaves its value
unchanged. Remove any existing tenant-controlled company.com/system labels
before enabling the platform-ingress allowance.
As Capsule we try to provide a secure multi-tenant environment out of the box, there are however some additional Admission Policies you should consider to enforce best practices in your cluster. Since Capsule only covers the core multi-tenancy features, such as Namespaces, Resource Quotas, Network Policies, and Container Registries, Classes, you should consider using an additional Admission Controller to enforce best practices on workloads and other resources.
Create custom Policies and reuse data provided via Tenant Status to enforce your own rules.
Let’s say we have the following namespaced ObjectBucketClaim resource:
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: admission-class
namespace: solar-production
finalizers:
- objectbucket.io/finalizer
labels:
bucket-provisioner: openshift-storage.ceph.rook.io-bucket
spec:
additionalConfig:
maxSize: 2G
bucketName: test-some-uid
generateBucketName: test
objectBucketName: obc-test-test
storageClassName: ocs-storagecluster-ceph-rgw
However since we are allowing Tenant Users to create these ObjectBucketClaims we might want to consider validating the storageClassName field to ensure that only allowed StorageClasses are used.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-tenant-class
spec:
validationFailureAction: Enforce
rules:
- name: restrict-storage-class
context:
- name: classes
apiCall:
urlPath: "/apis/capsule.clastix.io/v1beta2/tenants"
jmesPath: "items[?contains(status.namespaces, '{{ request.namespace }}')].status.classes | [0]"
- name: storageClass
variable:
jmesPath: "request.object.spec.storageClassName || 'NONE'"
match:
resources:
kinds:
- ObjectBucketClaim
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "storageclass {{ storageClass }} is not allowed in tenant ({{classes.storage}})"
deny:
conditions:
- key: "{{classes.storage}}"
operator: AnyNotIn
value: "{{ storageClass }}"Policies to harden workloads running in a multi-tenant environment.
Pod Disruption Budgets (PDBs) are a way to limit the number of concurrent disruptions to your Pods. In multi-tenant environments, it is recommended to enforce the usage of PDBs to ensure that tenants do not accidentally or maliciously block cluster operations.
A PodDisruptionBudget which sets its maxUnavailable value to zero prevents all voluntary evictions including Node drains which may impact maintenance tasks. This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field it must be greater than zero.
---
# Source: https://kyverno.io/policies/other/pdb-maxunavailable/pdb-maxunavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: pdb-maxunavailable
annotations:
policies.kyverno.io/title: PodDisruptionBudget maxUnavailable Non-Zero
policies.kyverno.io/category: Other
kyverno.io/kyverno-version: 1.9.0
kyverno.io/kubernetes-version: "1.24"
policies.kyverno.io/subject: PodDisruptionBudget
policies.kyverno.io/description: >-
A PodDisruptionBudget which sets its maxUnavailable value to zero prevents
all voluntary evictions including Node drains which may impact maintenance tasks.
This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field
it must be greater than zero.
spec:
validationFailureAction: Enforce
background: false
rules:
- name: pdb-maxunavailable
match:
any:
- resources:
kinds:
- PodDisruptionBudget
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "The value of maxUnavailable must be greater than zero."
pattern:
spec:
=(maxUnavailable): ">0"apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: pdb-maxunavailable
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["policy"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["poddisruptionbudgets"]
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validations:
- expression: |
!has(object.spec.maxUnavailable) ||
string(object.spec.maxUnavailable).contains('%') ||
object.spec.maxUnavailable > 0
message: "The value of maxUnavailable must be greater than zero or a percentage."
reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: pdb-maxunavailable-binding
spec:
policyName: pdb-maxunavailable
validationActions: ["Deny"]When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget, if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget it may prevent voluntary disruptions including Node drains which may impact routine maintenance tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have a matching PodDisruptionBudget to ensure these two values do not match.
---
# Source: https://kyverno.io/policies/other/pdb-minavailable/pdb-minavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: pdb-minavailable-check
annotations:
policies.kyverno.io/title: Check PodDisruptionBudget minAvailable
policies.kyverno.io/category: Other
kyverno.io/kyverno-version: 1.9.0
kyverno.io/kubernetes-version: "1.24"
policies.kyverno.io/subject: PodDisruptionBudget, Deployment, StatefulSet
policies.kyverno.io/description: >-
When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
it may prevent voluntary disruptions including Node drains which may impact routine maintenance
tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have
a matching PodDisruptionBudget to ensure these two values do not match.
spec:
validationFailureAction: Enforce
background: false
rules:
- name: pdb-minavailable
match:
any:
- resources:
kinds:
- Deployment
- StatefulSet
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
preconditions:
all:
- key: "{{`{{ request.operation | 'BACKGROUND' }}`}}"
operator: AnyIn
value:
- CREATE
- UPDATE
- key: "{{`{{ request.object.spec.replicas | '1' }}`}}"
operator: GreaterThan
value: 0
context:
- name: minavailable
apiCall:
urlPath: "/apis/policy/v1/namespaces/{{`{{ request.namespace }}`}}/poddisruptionbudgets"
jmesPath: "items[?label_match(spec.selector.matchLabels, `{{`{{ request.object.spec.template.metadata.labels }}`}}`)] | [0].spec.minAvailable | default(`0`)"
validate:
message: >-
The matching PodDisruptionBudget for this resource has its minAvailable value equal to the replica count
which is not permitted.
deny:
conditions:
any:
- key: "{{`{{ request.object.spec.replicas }}`}}"
operator: Equals
value: "{{`{{ minavailable }}`}}"PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.Introducing a PDB where there are already matching Pod controllers may pose a problem if the author is unaware of the existing replica count. This policy ensures that the minAvailable value is not greater or equal to the replica count of any matching existing Deployment. If other Pod controllers should also be included in this check, additional rules may be added to the policy which match those controllers.
---
# Source: https://kyverno.io/policies/other/deployment-replicas-higher-than-pdb/deployment-replicas-higher-than-pdb/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: deployment-replicas-higher-than-pdb
annotations:
policies.kyverno.io/title: Ensure Deployment Replicas Higher Than PodDisruptionBudget
policies.kyverno.io/category: Other
policies.kyverno.io/subject: PodDisruptionBudget, Deployment
kyverno.io/kyverno-version: 1.11.4
kyverno.io/kubernetes-version: "1.27"
policies.kyverno.io/description: >-
PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.
Introducing a PDB where there are already matching Pod controllers may pose a problem if the author
is unaware of the existing replica count. This policy ensures that the minAvailable value is not
greater or equal to the replica count of any matching existing Deployment. If other Pod controllers
should also be included in this check, additional rules may be added to the policy which match those
controllers.
spec:
validationFailureAction: Enforce
background: true
rules:
- name: deployment-replicas-greater-minAvailable
match:
any:
- resources:
kinds:
- PodDisruptionBudget
operations:
- CREATE
- UPDATE
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
context:
- name: deploymentreplicas
apiCall:
jmesPath: items[?label_match(`{{`{{ request.object.spec.selector.matchLabels }}`}}`, spec.template.metadata.labels)] || `[]`
urlPath: /apis/apps/v1/namespaces/{{`{{request.namespace}}`}}/deployments
preconditions:
all:
- key: '{{`{{ length(deploymentreplicas) }}`}}'
operator: GreaterThan
value: 0
- key: '{{`{{ request.object.spec.minAvailable || "" }}`}}'
operator: NotEquals
value: ''
validate:
message: >-
PodDisruption budget minAvailable ({{`{{ request.object.spec.minAvailable }}`}}) cannot be
greater than or equal to the replica count of any matching existing Deployment.
There are {{`{{ length(deploymentreplicas) }}`}} Deployments which match this labelSelector
having {{`{{ deploymentreplicas[*].spec.replicas }}`}} replicas.
foreach:
- list: deploymentreplicas
deny:
conditions:
all:
- key: "{{`{{ request.object.spec.minAvailable }}`}}"
operator: GreaterThanOrEquals
value: "{{`{{ element.spec.replicas }}`}}"When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget, if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget it may prevent voluntary disruptions including Node drains which may impact routine maintenance tasks and disrupt operations. This policy checks incoming CNPG Clusters and their .spec.enablePDB setting.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: pdb-cnpg-cluster-validation
annotations:
policies.kyverno.io/title: Check PodDisruptionBudget minAvailable for cnpgCluster
policies.kyverno.io/category: Other
kyverno.io/kyverno-version: 1.9.0
kyverno.io/kubernetes-version: "1.24"
policies.kyverno.io/subject: PodDisruptionBudget, Cluster
policies.kyverno.io/description: >-
When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
it may prevent voluntary disruptions including Node drains which may impact routine maintenance
tasks and disrupt operations. This policy checks incoming CNPG Clusters and their .spec.enablePDB setting.
spec:
validationFailureAction: Enforce
background: false
rules:
- name: pdb-cnpg-cluster-validation
match:
any:
- resources:
kinds:
- postgresql.cnpg.io/v1/Cluster
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
preconditions:
any:
- key: "{{request.operation || 'BACKGROUND'}}"
operator: AnyIn
value:
- CREATE
- UPDATE
validate:
message: >-
Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2.
deny:
conditions:
all:
- key: "{{request.object.spec.enablePDB }}"
operator: Equals
value: true
- key: "{{request.object.spec.instances }}"
operator: LessThan
value: 2apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: pdb-cnpg-cluster-validation
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["postgresql.cnpg.io"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["clusters"]
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validations:
- expression: |
!has(object.spec.enablePDB) ||
object.spec.enablePDB == false ||
(has(object.spec.instances) && object.spec.instances >= 2)
message: "Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2."
messageExpression: |
'Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2. Current instances: ' +
string(has(object.spec.instances) ? object.spec.instances : 1)
reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: pdb-cnpg-cluster-validation-binding
spec:
policyName: pdb-cnpg-cluster-validation
validationActions: ["Deny"]By Defaults emptyDir Volumes do not have any limits. This could lead to a situation, where a tenant fills up the node disk. To avoid this, you can enforce limits on emptyDir volumes. You may also consider restricting the usage of emptyDir with the medium: Memory option, as this could lead to memory exhaustion on the node.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tenant-workload-restrictions
spec:
rules:
- name: default-emptydir-sizelimit
match:
any:
- resources:
kinds:
- Pod
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
mutate:
foreach:
- list: "request.object.spec.volumes[]"
preconditions:
all:
- key: "{{element.keys(@)}}"
operator: AnyIn
value: emptyDir
- key: "{{element.emptyDir.sizeLimit || ''}}"
operator: Equals
value: ''
patchesJson6902: |-
- path: "/spec/volumes/{{elementIndex}}/emptyDir/sizeLimit"
op: add
value: 250Mi Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the kubectl debug functionality and attach a temporary container to an existing Pod. This may potentially be used to gain access to unauthorized information executing inside one or more containers in that Pod. This policy blocks the use of ephemeral containers.
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: block-ephemeral-containers
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
# 1) Regular Pods (ensure spec doesn't carry ephemeralContainers)
- apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
operations: ["CREATE","UPDATE"]
scope: "Namespaced"
# 2) Subresource used by `kubectl debug` to inject ephemeral containers
- apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods/ephemeralcontainers"]
operations: ["UPDATE","CREATE"] # UPDATE is typical, CREATE included for future-proofing
scope: "Namespaced"
validations:
# Deny any request that targets the pods/ephemeralcontainers subresource
- expression: request.subResource != "ephemeralcontainers"
message: "Ephemeral (debug) containers are not permitted (subresource)."
# For direct Pod create/update, allow only if the field is absent or empty
- expression: >
!has(object.spec.ephemeralContainers) ||
size(object.spec.ephemeralContainers) == 0
message: "Ephemeral (debug) containers are not permitted in Pod specs."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: block-ephemeral-containers-binding
spec:
policyName: block-ephemeral-containers
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists# Source: https://kyverno.io/policies/other/block-ephemeral-containers/block-ephemeral-containers/
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: block-ephemeral-containers
annotations:
policies.kyverno.io/title: Block Ephemeral Containers
policies.kyverno.io/category: Other
policies.kyverno.io/severity: medium
kyverno.io/kyverno-version: 1.6.0
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kubernetes-version: "1.23"
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the
`kubectl debug` functionality and attach a temporary container to an existing Pod.
This may potentially be used to gain access to unauthorized information executing inside
one or more containers in that Pod. This policy blocks the use of ephemeral containers.
spec:
validationFailureAction: Enforce
background: true
rules:
- name: block-ephemeral-containers
match:
any:
- resources:
kinds:
- Pod
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "Ephemeral (debug) containers are not permitted."
pattern:
spec:
X(ephemeralContainers): "null"Often when working in multi-tenant environments, you want to ensure that tenants are not using ClusterIssuers to issue certificates, but rather use namespaced Issuers within their own namespace. This policy enforces that cert-manager.io/v1/Certificate resources do not reference ClusterIssuers and that the Issuer referenced is in the same namespace as the Certificate.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: certificates-only-local-issuer
spec:
validationFailureAction: Enforce
background: true
rules:
- name: deny-clusterissuer-in-certificates
match:
any:
- resources:
kinds:
- cert-manager.io/v1/Certificate
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "Certificates must not reference ClusterIssuers; use a namespaced Issuer in the same namespace."
deny:
conditions:
any:
- key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
operator: Equals
value: "ClusterIssuer"
- name: deny-cross-namespace-issuerref-in-certificates
match:
any:
- resources:
kinds:
- cert-manager.io/v1/Certificate
validate:
message: "Certificates must reference an Issuer in the same namespace (spec.issuerRef.namespace must be empty or equal to the Certificate namespace)."
deny:
conditions:
any:
# If issuerRef.namespace is set and differs from the Certificate namespace -> deny
- key: "{{request.object.spec.issuerRef.namespace || '' }}"
operator: NotEquals
value: ""
# AND also not equal to request namespace
- key: "{{ request.object.spec.issuerRef.namespace || request.namespace }}"
operator: NotEquals
value: "{{ request.namespace }}"Deny to usage of ClusterIssuers in Gateways by checking for the cert-manager.io/cluster-issuer annotation. This ensures that tenants use namespaced issuer mechanisms instead.
This requires extra permissions to allow Kyverno to read Gateway resources:
admissionController:
rbac:
clusterRole:
extraResources:
- apiGroups: ["gateway.networking.k8s.io"]
resources: ["*"]
verbs: ["get", "list", "watch"]
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: gateways-deny-cluster-issuer-annotation
spec:
validationFailureAction: Enforce
background: false
rules:
- name: deny-cert-manager-cluster-issuer-annotation
match:
any:
- resources:
kinds:
- gateway.networking.k8s.io/v1/Gateway
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "Gateways must not use cert-manager.io/cluster-issuer; use namespaced issuer mechanisms instead."
deny:
conditions:
any:
- key: "{{ request.object.metadata.annotations.\"cert-manager.io/cluster-issuer\" || '' }}"
operator: NotEquals
value: ""Allow certain ClusterIssuers within Tenants:
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: certificates-restrict-clusterissuer-by-tenant-label
spec:
validationFailureAction: Enforce
background: true
rules:
- name: allow-clusterissuer-only-when-managed-by-matches
match:
any:
- resources:
kinds:
- cert-manager.io/v1/Certificate
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
context:
- name: certManagedBy
variable:
jmesPath: request.object.metadata.labels."capsule.clastix.io/managed-by" || ''
- name: clusterIssuerManagedBy
apiCall:
urlPath: /apis/cert-manager.io/v1/clusterissuers/{{ request.object.spec.issuerRef.name }}
jmesPath: metadata.labels."company.com/tenant" || ''
default: ""
preconditions:
all:
- key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
operator: Equals
value: ClusterIssuer
- key: "{{request.operation || 'BACKGROUND'}}"
operator: AnyIn
value:
- CREATE
- UPDATE
validate:
message: >-
ClusterIssuer is only allowed when the Certificate label
capsule.clastix.io/managed-by ({{certManagedBy}}) matches the referenced ClusterIssuer label
company.com/tenant ({{clusterIssuerManagedBy}}).
deny:
conditions:
any:
- key: "{{certManagedBy}}"
operator: Equals
value: ""
- key: "{{clusterIssuerManagedBy}}"
operator: Equals
value: ""
- key: "{{certManagedBy}}"
operator: NotEquals
value: "{{clusterIssuerManagedBy}}"Concept references explain reusable behavior shared across Capsule APIs:
Introducing a new separation of duties can lead to a significant paradigm shift. This has technical implications and may also impact your organizational structure. Therefore, when designing a multi-tenant platform pattern, carefully consider the following aspects. As Cluster Administrator, ask yourself:
The answer to this question may be influenced by the following aspects:
Are the Cluster Administrators willing to grant permissions to Tenant Owners?
Who is responsible for the deployed workloads within the Tenants?
Who gets paged during a production outage within a Tenant’s application?
Are your customers technically capable of working directly with the Kubernetes API?
In Capsule, we introduce a new persona called the Tenant Owner. The goal is to enable Cluster Administrators to delegate tenant management responsibilities to Tenant Owners. Here’s how it works:
Configure Capsule Administrators. The ClusterRoles assigned to Administrators can be configured in the CapsuleConfiguration as well.
They are promoted to Tenant-Owners for all available tenants. Effectively granting them the ability to manage all namespaces within the cluster, across all tenants.
Note: Granting Capsule Administrator rights should be done with caution, as it provides extensive control over the cluster’s multi-tenant environment. When granting Capsule Administrator rights, the entity gets the privileges to create any namespace (also not part of capsule tenants) and the privileges to delete any tenant namespaces.
Capsule Administrators can:
Administrators come in handy in bootstrap scenarios or GitOps scenarios where certain users/serviceaccounts need to be able to manage namespaces for all tenants.
Any entity which needs to interact with tenants and their namespaces must be defined as a Capsule User. This is where the flexibility of Capsule comes into play. You can define users or groups as Capsule Users, allowing them to create and manage namespaces within any tenant they have access to. If they are not defined as Capsule Users, any interactions will be ignored by Capsule. Often a best practice is to define a single group which identifies all your tenant users. This way you can have one generic group for all your users and then use additional groups to separate responsibilities (e.g. administrators vs normal users).
Only one entry is needed to identify a Capsule User. This is only important for Namespace Admission..
You can always verify the effective Capsule Users by checking the Configuration Status. As this is variable with Tenant Owners, the status will always show the effective users:
kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq
[
{
"kind": "Group",
"name": "oidc:kubernetes:admin"
},
{
"kind": "Group",
"name": "projectcapsule.dev"
},
{
"kind": "User",
"name": "test-user"
}
]
Every Tenant Owner must be a Capsule User. By using the TenantOwner CRD this is automatically handeled.
They manage the namespaces within their tenants and perform administrative tasks confined to their tenant boundaries. This delegation allows teams to operate more autonomously while still adhering to organizational policies. Tenant Owners can be used to shift reposnsability of one tenant towards this user group. promoting them to the SPOC of all namespaces within the tenant.
Tenant Owners can:
Capsule provides robust tools to strictly enforce tenant boundaries, ensuring that each tenant operates within its defined limits. This separation of duties promotes both security and efficient resource management.
Let’s discuss different Tenant Layouts which could be used . These are just approaches we have seen, however you might also find a combination of these which fits your use-case.
With this approach you essentially just provide your Customers with the Tenant on your cluster. The rest is their responsibility. This concludes to a shared responsibility model. This can be achieved when also the Tenant Owners are responsible for everything they are provisiong within their Tenant’s namespaces.
Workload distribution across your compute infrastructure can be approached in various ways, depending on your specific priorities. Regardless of the use case, it’s essential to preserve maximum flexibility for your platform administrators. This means ensuring that:
If your cluster architecture prevents any of these capabilities, or if certain applications block the enforcement of these policies, you should reconsider your approach.
Strong tenant isolation, ensuring that any noisy neighbor effects remain confined within individual tenants (tenant responsibility). This approach may involve higher administrative overhead and costs compared to shared compute. It also provides enhanced security by dedicating nodes to a single customer/application. It is recommended, at a minimum, to separate the cluster’s operator workload from customer workloads.
With this approach you share the nodes amongst all Tenants, therefore giving you more potential for optimizing resources on a node level. It’s a common pattern to separate the controllers needed to power your Distribution (operators) from the actual workload. This ensures smooth operations for the cluster
Overview:
We provide the concept of ResourcePools or CustomQuotas to manage resources cross namespaces. There’s some further aspects you must think about with shared approaches:
Capsule parameter schemas define the contract between a template and the values
submitted by a consuming API object. Resource Permits currently use spec.paramSchema
to validate ResourcePermit.spec.params before context loading and rendering.
spec.paramSchema on the referenced template is the complete contract for
ResourcePermit.spec.params. It accepts JSON Schema 2020-12 keywords,
Kubernetes-compatible x-kubernetes-validations CEL rules, and Capsule’s
x-capsule-form presentation extension.
Capsule validates parameters during ResourcePermit admission, before loading context or
rendering resources. Invalid parameters are rejected and no ResourcePermit is created.
Failures that require runtime context or rendering remain visible on an accepted
request as Created with Ready=False; Capsule does not apply any resources.
Use ordinary JSON Schema for types, required fields, enums, ranges, lengths, patterns, and conditional structure:
paramSchema:
$schema: https://json-schema.org/draft/2020-12/schema
type: object
additionalProperties: false
required:
- subjectKind
- subjectName
properties:
subjectKind:
type: string
enum: [User, Group, ServiceAccount]
subjectName:
type: string
minLength: 1
allOf:
- if:
properties:
subjectKind:
const: ServiceAccount
required: [subjectKind]
then:
properties:
subjectName:
pattern: '^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?/[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$'
With this schema:
ServiceAccount with operations/resource-permit-runner is valid;ServiceAccount with resource-permit-runner is invalid; andUser with alice@example.com remains valid.The example represents a ServiceAccount as namespace/name. When rendering an RBAC
Subject, split that value into the subject’s namespace and name fields.
Other useful JSON Schema keywords include dependentRequired,
dependentSchemas, contains, prefixItems, allOf, anyOf, oneOf,
and unevaluatedProperties.
Use x-kubernetes-validations for readable cross-field and collection invariants.
It has the same shape as validation rules generated from Kubebuilder’s
+kubebuilder:validation:XValidation marker. Capsule converts the parameter schema
to a Kubernetes structural schema and delegates evaluation to Kubernetes’ CEL
validator.
paramSchema:
type: object
additionalProperties: false
required:
- subjectKind
- subjectName
properties:
subjectKind:
type: string
enum: [User, Group, ServiceAccount]
subjectName:
type: string
minLength: 1
x-kubernetes-validations:
- rule: >-
self.subjectKind != 'ServiceAccount' ||
self.subjectName.matches('^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?/[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$')
message: ServiceAccount subjects must use namespace/name
reason: FieldValueInvalid
fieldPath: .subjectName
self is the value at the schema node containing the extension. At the root it is
the whole parameters object. Use has(self.field) for optional properties.
The accepted validation-rule fields are:
| Field | Meaning |
|---|---|
rule | Required CEL expression returning a boolean |
message | Static failure message; required when a rule contains line breaks |
messageExpression | CEL expression returning a dynamic failure string |
reason | FieldValueInvalid, FieldValueForbidden, FieldValueRequired, or FieldValueDuplicate |
fieldPath | Relative field path associated with the error |
optionalOldSelf | Kubernetes transition-rule behavior for an optional previous value |
Rule and message-expression syntax is compiled when the template is admitted. Evaluation uses Kubernetes CEL cost limits.
Parameter validation has no previous parameter object. A transition rule using
oldSelf therefore behaves like Kubernetes validation on create: an ordinary
transition rule is skipped, while optionalOldSelf: true can explicitly handle an
absent previous value. Use self for request-time constraints.
Dynamic failure messages are supported:
x-kubernetes-validations:
- rule: "self.subjectKind != 'ServiceAccount' || self.subjectName.contains('/')"
message: invalid ServiceAccount path
messageExpression: >-
'subjectName ' + self.subjectName + ' must use namespace/name for a ServiceAccount'
reason: FieldValueInvalid
fieldPath: .subjectName
If messageExpression fails or produces an invalid message, evaluation falls back
to message, then to the rule text.
Rules may also be attached to structural properties, array items, and
additional-property schemas. Their self value is scoped accordingly:
paramSchema:
type: object
properties:
subjects:
type: array
items:
type: object
required: [name]
properties:
name:
type: string
x-kubernetes-validations:
- rule: "!self.name.startsWith('system:')"
messageExpression: "'subject ' + self.name + ' uses a reserved prefix'"
fieldPath: .name
CEL rules must be attached to Kubernetes structural schema nodes. Do not place them
inside $defs, if/then, or another JSON-Schema-only branch. Put the rule on
a root, property, item, or additional-property node that can see the required values.
Standard JSON Schema may still use those constructs alongside CEL.
Choose the smallest validation mechanism that expresses the invariant:
| Requirement | Mechanism |
|---|---|
| Type, enum, range, length, or regex | JSON Schema property keyword |
| Field required for a discriminator value | JSON Schema if/then |
| Relationship between fields | x-kubernetes-validations CEL |
| Collection-item relationship | Nested CEL rule or CEL collection macro |
| Populate choices from live objects | x-capsule-form |
| Authoritatively load a selected object | Template context.resources |
x-capsule-form is an optional JSON Schema vendor extension for compatible clients
such as a Headlamp plugin. It populates a string field from live Kubernetes objects
without changing server-side JSON Schema semantics.
This complete example combines JSON Schema, conditional validation, CEL, and a live Secret selector:
paramSchema:
$schema: https://json-schema.org/draft/2020-12/schema
type: object
additionalProperties: false
required:
- subjectKind
- subjectName
- credentialsSecret
properties:
subjectKind:
type: string
enum: [User, Group, ServiceAccount]
subjectName:
type: string
minLength: 1
credentialsSecret:
type: string
pattern: '^[a-z0-9.-]+/[a-z0-9.-]+$'
x-capsule-form:
widget: kubernetes-resource
source:
apiVersion: v1
kind: Secret
namespace: request
labelSelector: access.example.com/selectable=true
option:
labelTemplate: '{{ .metadata.name }}'
valueTemplate: '{{ .metadata.namespace }}/{{ .metadata.name }}'
allOf:
- if:
properties:
subjectKind:
const: ServiceAccount
required: [subjectKind]
then:
properties:
subjectName:
pattern: '^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?/[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$'
x-kubernetes-validations:
- rule: >-
self.subjectKind != 'Group' ||
!self.subjectName.startsWith('system:')
message: Groups using the system prefix cannot be requested
reason: FieldValueForbidden
fieldPath: .subjectName
The accepted form-extension shape is:
x-capsule-form:
widget: kubernetes-resource # required
source: # required
apiVersion: group.example.io/v1 # required; v1 is also valid
kind: Example # required; one exact kind
namespace: request # optional: request, "*", or a namespace
labelSelector: key=value # optional
fieldSelector: metadata.name=x # optional
option: # optional
labelTemplate: '{{ .metadata.name }}'
valueTemplate: '{{ .metadata.namespace }}/{{ .metadata.name }}'
Unknown widgets and extension fields are rejected. option may be omitted; both
templates then default to {{ .metadata.name }}.
| Field | Required | Meaning |
|---|---|---|
apiVersion | Yes | Exact version such as v1, apps/v1, or an arbitrary CRD version |
kind | Yes | Exact Kubernetes kind; wildcards are unsupported |
namespace | No | List scope for a namespaced GVK |
labelSelector | No | Kubernetes label selector passed to the list request |
fieldSelector | No | Kubernetes field selector passed to the list request |
namespace supports four modes:
| Value | Behavior |
|---|---|
| omitted | Request namespace for namespaced GVKs; cluster endpoint for cluster-scoped GVKs |
request | Explicitly use the ResourcePermit namespace |
* | List a namespaced GVK across every namespace |
| namespace name | List only that literal namespace |
Any exact discoverable GVK is supported, including CRDs. Capsule validates API-version syntax, selectors, namespaces, extension fields, and option-template syntax when the template is admitted. It does not require the GVK to exist at that time, allowing the template to be installed before its CRD.
The form client uses Kubernetes discovery to determine whether a GVK is namespaced or cluster-scoped. For example:
properties:
gatewayClass:
type: string
x-capsule-form:
widget: kubernetes-resource
source:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
option:
labelTemplate: '{{ .metadata.name }} — {{ .spec.controllerName }}'
valueTemplate: '{{ .metadata.name }}'
To list a namespaced type across all namespaces:
properties:
sourceConfigMap:
type: string
x-capsule-form:
widget: kubernetes-resource
source:
apiVersion: v1
kind: ConfigMap
namespace: '*'
labelSelector: access.example.com/exportable=true
option:
valueTemplate: '{{ .metadata.namespace }}/{{ .metadata.name }}'
The user’s Kubernetes RBAC must permit the corresponding list operation.
The listed object is an unstructured map at the Go-template root:
option:
labelTemplate: '{{ .metadata.name }} ({{ .metadata.namespace }})'
valueTemplate: '{{ .metadata.namespace }}/{{ .metadata.name }}'
Both fields default to {{ .metadata.name }}. They use Capsule’s safe Go-template
function map; environment-reading functions are unavailable. Keep values deterministic
because the rendered value is stored in spec.params.
Capsule finds and validates x-capsule-form recursively in properties, array items,
tuple and prefix items, $defs, definitions, allOf, anyOf, oneOf, and
conditional schemas:
paramSchema:
type: object
properties:
targets:
type: array
items:
type: object
properties:
secret:
type: string
x-capsule-form:
widget: kubernetes-resource
source:
apiVersion: v1
kind: Secret
namespace: request
For one or more selections, define an array and place x-capsule-form on its string
items schema. A compatible client may display this as a multi-select control or as
repeatable resource selectors:
properties:
clusterRoles:
type: array
minItems: 1
uniqueItems: true
items:
type: string
x-capsule-form:
widget: kubernetes-resource
source:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
namespaces:
type: array
minItems: 1
uniqueItems: true
items:
type: string
x-capsule-form:
widget: kubernetes-resource
source:
apiVersion: v1
kind: Namespace
minItems enforces at least one selection and uniqueItems prevents duplicate
generator output. The submitted values remain ordinary arrays in spec.params and
can be iterated with nested Go-template range actions.
A compatible client:
x-capsule-form;apiVersion and kind through Kubernetes discovery;spec.params.Discovery and listing must use the logged-in user’s credentials, not the template’s execution ServiceAccount. Otherwise a dropdown could disclose objects the user cannot list.
x-capsule-form is presentation metadata, not an authorization boundary:
Validate submitted values with JSON Schema or CEL. For authoritative lookup, load the
selected object through spec.context.resources, which uses the recorded template
ServiceAccount:
paramSchema:
type: object
required: [crd]
properties:
crd:
type: string
x-capsule-form:
widget: kubernetes-resource
source:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
context:
resources:
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: '{{ .crd }}'
index: crdInfo
optional: false
If the context object is missing or the execution identity cannot read it, the request
becomes Ready=False.
Capsule uses a shared resource manager to apply Kubernetes objects, track their
ownership, and prune them when they leave scope. TenantResource,
GlobalTenantResource, ResourcePermitTemplate, and
GlobalResourcePermitTemplate share spec.resources[].policy. Each block controls
its generated resources, allowing namespaces within a tenant to receive different
configuration and lifecycle behavior.
Capsule uses Kubernetes server-side apply (SSA) with a stable field manager for each managing parent. A ResourcePermit manager is derived from the request UID:
projectcapsule.dev/resource/resourcepermit/<request-uid>
SSA allows Capsule and another manager to own different fields on the same object. When both managers claim the same field, Kubernetes reports an apply conflict unless Capsule is configured to force ownership.
For Resource Permits, Capsule renders every target before approval and stores the concrete
manifests and their policies in .status.request.resources. That snapshot—not the
mutable source template—is the source of truth for review, apply, and cleanup.
One policy applies to every target in a resource group:
resources:
- policy:
creation: Owner
protect: true
force: false
deletion: Remove
targets:
- apiVersion: v1
kind: ConfigMap
metadata:
name: temporary-settings
| Field | Values | Default | Behavior |
|---|---|---|---|
creation | Owner, Merge | Owner | Whether an existing object may be adopted |
protect | true, false | true | Whether admission blocks direct changes while managed |
force | true, false | false | Whether SSA may take conflicting field ownership |
deletion | Remove, Orphan | Remove | What happens when the parent stops managing the object |
condition | CEL boolean expression | Omitted | Whether to apply rendered content to each destination |
Owner requires Capsule to have created the target. If an object with the same
identity already exists and was not created for the managing API, reconciliation
fails. Merge adopts an existing object when possible and creates it otherwise. Use
Merge only when the workflow intentionally manages fields on pre-existing objects.
force: true resolves SSA conflicts by taking ownership of the rendered fields. It
does not bypass Kubernetes RBAC, admission, or schema validation, and can displace
another controller’s ownership.
With protect: true, Capsule marks the object as protected. The generic Capsule
admission webhook rejects direct updates and deletion while the parent manages it,
except for the controller and the resolved execution ServiceAccount. Disable
protection only when another actor must modify the object during its managed lifetime.
With deletion: Remove, when multiple Capsule resource managers share an adopted
target, removing one manager preserves the shared tracking and protection metadata while another
Capsule resource manager remains. The final departing manager removes that
metadata without deleting fields owned by external managers.
Cleanup depends on both the deletion policy and whether Capsule created or adopted the object:
| Policy | Object created by Capsule | Existing object adopted with Merge |
|---|---|---|
Remove | Delete the object | Relinquish managed fields and tracking |
Orphan | Keep the object and remove tracking/protection | Keep the object and applied fields, then remove tracking/protection |
Orphan can leave privileges or configuration behind after the parent expires, so it
should be an explicit design choice.
Set spec.resources[].policy.condition on TenantResource,
GlobalTenantResource, ResourcePermitTemplate, or
GlobalResourcePermitTemplate. One expression applies independently to every
destination produced by that resource block; separate blocks can use different
conditions. The expression must return a boolean and may contain up to 4096
characters. Omit the field to keep unconditional apply; an empty or whitespace-only
expression is not a substitute for omitting it.
| Input | Meaning |
|---|---|
object | The existing destination resource, read using the execution identity; null if it does not exist. It is not the rendered candidate. |
now | The UTC timestamp at evaluation time. Use timestamp(...) and duration(...) for time comparisons. |
Apply conditions use CEL directly. There is no self, .self, Go-template
expansion, or access to template parameters/context in this expression. Load
context and render the desired resource through the existing template API. The
condition then decides whether that rendered resource may be applied.
For example, create a target only when it is absent:
policy:
condition: "object == null"
To initialize a target and then allow a refresh after five minutes:
policy:
condition: |
object == null ||
!has(object.metadata.annotations) ||
!('keys.example.org/rotated-at' in object.metadata.annotations) ||
now >= timestamp(object.metadata.annotations['keys.example.org/rotated-at']) + duration('5m')
The generator must write the keys.example.org/rotated-at annotation on each
successful refresh. Guard missing objects and fields before accessing them;
use bracket notation for annotation keys containing dots or slashes. A missing
timestamp initializes this example, while a malformed timestamp causes an
evaluation error. See the complete rotation example
for the template, retention behavior, and verification steps.
| Outcome | Apply behavior |
|---|---|
| Condition omitted | Existing unconditional apply behavior. |
true | Apply using the block’s creation, force, and protection policies; normal authorization and ownership checks still apply. |
false | Skip rendered content and preserve its last-apply timestamp. Reconcile protection and retain the current policy for already managed targets. A never-applied target is not adopted, protected, or modified. |
| Invalid syntax or non-boolean result type | Admission rejects the expression at spec.resources[i].policy.condition. |
| Evaluation or target-read error | The target is not written; the failure is reported through existing status. A forbidden read is not treated as an absent object. |
There is no configurable action for a false result. Conditions do not provide an all-or-nothing transaction across targets: each target is evaluated separately. They also do not change SSA retention. To preserve old fields owned by Capsule, include them in subsequent rendered resources, as the age-key example does.
For example, changing protect: true to protect: false removes protection on
an already managed target even while its condition is false, provided no other
Capsule resource manager still shares the target. Protection is also reconciled
when the condition becomes true or is removed; protection established during a
skipped apply does not persist after a sole manager disables it. Changing
deletion: Remove to deletion: Orphan takes effect for subsequent cleanup.
creation and force govern the next permitted content apply; changing them
does not override a false condition. Rendered labels, annotations, and data
remain unchanged, including key material and rotation timestamps. Capsule may
patch its own protection metadata, so the target’s resource version can change.
When that metadata already matches, a skipped reconciliation performs no target
write. Policy changes that fail to reconcile are reported through existing status.
| Resource | Evaluation lifecycle | Suitable use |
|---|---|---|
TenantResource / GlobalTenantResource | Render and evaluate during reconciliation, including periodic resync. | Recurring namespace configuration and key rotation. |
ResourcePermitTemplate / GlobalResourcePermitTemplate | Render a request snapshot, then evaluate against live targets during preflight, activation, and explicit failure retries. Active permits do not periodically reevaluate. | Conditional provisioning during an approved permit’s activation. |
Permit approval conditions (spec.approvals.conditions) control approval
eligibility. Apply conditions control target writes after those approval rules
are satisfied. In particular, an Active permit may contain skipped targets;
use approval conditions when the entire permit must be gated. The
permit guide
provides namespaced/global examples, required permissions, and status diagnostics.
For namespaced targets, Capsule normally defaults an omitted namespace to the parent workflow’s namespace. An explicitly rendered namespace is preserved when the workflow is trusted to operate across namespaces. Cluster-scoped GVKs are applied without a namespace.
A namespaced parent cannot own a cluster-scoped object through a Kubernetes owner reference. APIs such as ResourcePermit therefore record every processed target and use a finalizer for explicit cascading cleanup. Successful targets remain recorded even if a later target fails, allowing retries and finalization to handle partial application.
Managed objects may carry labels and annotations such as:
metadata:
labels:
projectcapsule.dev/created-by: resource-permit
projectcapsule.dev/managed-by: resource-permit
projectcapsule.dev/protected-by: resource-permit
annotations:
projectcapsule.dev/active-until: "2026-08-31T11:00:00Z"
projectcapsule.dev/resourcepermit-service-account: system:serviceaccount:capsule-system:resource-permit-runner
The exact metadata depends on the parent API, policy, and lifecycle stage. Capsule sets tracking metadata independently of the rendered manifest, so a template cannot forge whether an object was created or adopted.
Managed APIs use a shared processed-resource status shape. A ResourcePermit example is:
status:
size: 2
processedItems:
- group: rbac.authorization.k8s.io
version: v1
kind: Role
namespace: solar-uat
name: incident-1042-editor
status:
clusterScoped: false
created: true
lastApply: "2026-08-31T10:02:13Z"
type: Ready
status: "True"
- group: rbac.authorization.k8s.io
version: v1
kind: ClusterRole
name: incident-1042-cluster-reader
status:
clusterScoped: true
created: true
lastApply: "2026-08-31T10:02:14Z"
type: Ready
status: "True"
Each item identifies its GVK, name, namespace and scope, whether it was created or
adopted, its last apply time, and its latest condition or error. Skipped targets
report ConditionNotMet: apply skipped. A previously applied target keeps its last
successful apply time; a never-applied target has none. The retained policy records
the last successful policy reconciliation and determines subsequent cleanup. Ownership violations,
SSA conflicts, RBAC denials, unavailable GVKs, invalid manifests, and admission
failures are reported through this status and the parent conditions.
kubectl get resourcepermit REQUEST -n NAMESPACE -o json \
| jq '.status | {size, processedItems, conditions}'
Do not edit controller-owned rendered manifests or processed-item status to recover from a failure. Correct the underlying object, permissions, template, or policy and let reconciliation retry.
Labels commonly used in Capsule items are listed below. These labels are applied to the corresponding resources by the Capsule controller and can be used for filtering and selection purposes.
capsule.clastix.io/tenant| Description | Target Objects | Audience |
|---|---|---|
Established the connection between Object and Tenant. It’s value indicates the owning Tenant | * Namespaces having a relationship to Tenants | Controller |
projectcapsule.dev/tenant| Description | Target Objects | Audience |
|---|---|---|
Established the connection between Object and Tenant. It’s value indicates the owning Tenant. Long term replacement for capsule.clastix.io/tenant and capsule.clastix.io/managed-by labels. | * All namespaced items within a Tenant Namespace become the corresponding Tenant Label via Mutating Admission. | Controller |
capsule.clastix.io/managed-by| Description | Target Objects | Audience |
|---|---|---|
Established the connection between Object and Tenant. It’s value indicates the owning Tenant. Long term replacement for capsule.clastix.io/tenant | * All namespaced items within a Tenant Namespace become the corresponding Tenant Label via Mutating Admission. This label is still added to keep compatibility wiht the Capsule Proxy. | User |
projectcapsule.dev/managed-by| Description | Target Objects | |
|---|---|---|
Indicator which controller of capsule or Custom Resource is responsible for managing the corresponding Object. Mainly used in Replications to establish that objects are at least managed by one Replications. | * Any Object being influenced by Replications | User |
projectcapsule.dev/created-by| Description | Target Objects | Audience |
|—|—|
| Indicator which controller of capsule or Custom Resource is responsible for managing the corresponding Object. Mainly used in Replications to establish that objects were originally created by a Replication. | * Any Object being influenced by Replications | Controller |
projectcapsule.dev/name| Description | Target Objects | Audience |
|---|---|---|
| Label for tracking internal name or allowing for faster selects. Mainly used to identify relevant rulestatus | * Tenant Namespaces | Controller |
projectcapsule.dev/cordoned| Description | Target Objects | Audience |
|---|---|---|
Indicator that a namespace is cordoned (when value equals true) | * Tenant Namespaces | User |
projectcapsule.dev/pool| Description | Target Objects | Audience |
|---|---|---|
| Allocation of Resourcepool via ResourcePoolClaims | * ResourcePoolClaims | User |
For simple template cases we provide a fast templating engine. With this engine, you can use Go templates syntax to reference Tenant and Namespace fields. There are no operators or anything else supported.
Available fields are:
{{tenant.name}}: The Name of the Tenant{{namespace}}: The Name of the namespace within the tenant (current context)Our template library is mainly based on the upstream implementation from Sprout. You can find the all available functions here:
We have removed certain functions which could exploit runtime information. Therefor the following functions are not available:
envexpandEnvYou can provide structured data for each Tenant which can be used in templating:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
data:
bool: true
foo: bar
list:
- a
- b
number: 123
obj:
nested: value
Templates rendered by a TenantResource or GlobalTenantResource always have the metadata of the replication object available under $.replications.metadata. An explicit resources[].context is not required.
Commonly used fields include:
$.replications.metadata.name$.replications.metadata.namespace for a namespaced TenantResource$.replications.metadata.labels$.replications.metadata.annotations$.replications.metadata.uid$.replications.metadata.resourceVersion$.replications.metadata.generation$.replications.metadata.ownerReferences$.replications.metadata.finalizersFor example:
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ $.replications.metadata.name }}
labels:
replication-name: {{ $.replications.metadata.name | quote }}
team: {{ index $.replications.metadata.labels "company.example/team" | quote }}
data:
sourceNamespace: {{ $.replications.metadata.namespace | quote }}
metadata.namespace is absent for a cluster-scoped GlobalTenantResource. The context excludes managedFields and the kubectl.kubernetes.io/last-applied-configuration annotation.
Custom Functions we provide in our template package.
getResourceByName finds a Kubernetes object by metadata.name in a resource collection loaded through resources[].context.resources[].index.
getResourceByName(name string, resources any) map[string]any
The function is pipe-friendly. The indexed resource collection is passed last:
{{- $management := $.mgmt | getResourceByName "tenant-management" -}}
If no object matches, the function returns an empty map. This makes it suitable for optional resources:
{{- with ($.mgmt | getResourceByName "optional-settings") }}
data:
source: {{ .metadata.name | quote }}
{{- end }}
If multiple objects have the requested name, rendering fails because the match is ambiguous. Use getResourceByNamespacedName when an index can contain objects from multiple namespaces.
mustGetResourceByName is the required variant of getResourceByName. Rendering fails if no object matches or if multiple objects have the requested name.
mustGetResourceByName(name string, resources any) map[string]any
Example using a ConfigMap value containing YAML:
{{- $management := $.mgmt | mustGetResourceByName "tenant-management" -}}
{{- $team := $management.data | get "team-a" | fromYAML -}}
subjects:
{{ $team.subjects | toYAML | indent 2 }}
getResourceByNamespacedName finds an object by both metadata.namespace and metadata.name. Use it when a context index contains resources from more than one namespace.
getResourceByNamespacedName(namespace string, name string, resources any) map[string]any
The namespace and name are supplied before the piped resource collection:
{{- $management := $.mgmt | getResourceByNamespacedName "solar-system" "tenant-management" -}}
If no object matches, the function returns an empty map. Rendering fails if the namespace and name still identify multiple objects.
deterministicUUID generates a deterministic, RFC-4122–compliant UUID (version 5 + RFC4122 variant) from a set of input strings. It is designed for use in templates where you need stable, repeatable IDs derived from meaningful inputs (e.g. cluster name, tenant, role name), instead of random UUIDs.
This is especially useful for:
The function takes any number of strings and turns them into a UUID in a fully deterministic way.
What that means in practice:
So from the outside, it behaves just like a normal UUID, just deterministic.
deterministicUUID(parts ...string) string
Example usage:
{{ deterministicUUID "cluster-a" "app-123" "tenant-x" "some-role" }}
generateAgeKey generates a new age X25519 key pair for use with age. It returns both the private identity and the public recipient key.
This is useful in templates where a resource needs to create an age-compatible encryption identity, for example when generating Kubernetes Secrets that are later used for encrypting or decrypting data.
This is especially useful for:
The function does not return a plain string. It returns an object with two fields:
Identity: the private age identity, e.g. AGE-SECRET-KEY-1...Recipient: the public age recipient, e.g. age1...What that means in practice:
generateAgeKey() any
Example usage:
{{ $key := generateAgeKey }}
apiVersion: v1
kind: Secret
metadata:
name: age-key
type: Opaque
stringData:
identity: {{ $key.Identity | quote }}
recipient: {{ $key.Recipient | quote }}
Output:
apiVersion: v1
kind: Secret
metadata:
name: age-key
type: Opaque
stringData:
identity: "AGE-SECRET-KEY-1..."
recipient: "age1..."
Because this function creates a new random key pair on every call, it should usually only be used when the generated Secret is created once and then reused. For continuously reconciled resources, prefer generating the key in controller logic and persisting it before using it in templates.
generateAgePQKey generates a new age post-quantum hybrid key pair for use with age. It returns both the private identity and the public recipient key.
This is useful in templates where a resource needs to create an age-compatible encryption identity using the newer hybrid recipient format.
This is especially useful for:
The function does not return a plain string. It returns an object with two fields:
Identity: the private age hybrid identity, e.g. AGE-SECRET-KEY-PQ-1...Recipient: the public age recipient, e.g. age1...What that means in practice:
generateAgePQKey() any
Example Usage:
{{ $key := generateAgePQKey }}
apiVersion: v1
kind: Secret
metadata:
name: age-pq-key
type: Opaque
stringData:
identity: {{ $key.Identity | quote }}
recipient: {{ $key.Recipient | quote }}
Output:
apiVersion: v1
kind: Secret
metadata:
name: age-pq-key
type: Opaque
stringData:
identity: "AGE-SECRET-KEY-PQ-1..."
recipient: "age1..."
Because this function creates a new random key pair on every call, it should usually only be used when the generated Secret is created once and then reused. For continuously reconciled resources, prefer generating the key in controller logic and persisting it before using it in templates.
Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by userGroups option in the CapsuleConfiguration, which defaults to projectcapsule.dev.
In the following guide, we’ll use Keycloak an Open Source Identity and Access Management server capable to authenticate users via OIDC and release JWT tokens as proof of authentication.
Configure Keycloak as OIDC server:
projectcapsule.devprojectcapsule.devkubernetes (Public)kubernetes-auth (Confidential (Client Secret))For the kubernetes client, create protocol mappers called groups and audience If everything is done correctly, now you should be able to authenticate in Keycloak and see user groups in JWT tokens. Use the following snippet to authenticate in Keycloak as alice user:
$ KEYCLOAK=sso.clastix.io
$ REALM=kubernetes-auth
$ OIDC_ISSUER=${KEYCLOAK}/realms/${REALM}
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/token \
-d grant_type=password \
-d response_type=id_token \
-d scope=openid \
-d client_id=${OIDC_CLIENT_ID} \
-d client_secret=${OIDC_CLIENT_SECRET} \
-d username=${USERNAME} \
-d password=${PASSWORD} | jq
The result will include an ACCESS_TOKEN, a REFRESH_TOKEN, and an ID_TOKEN. The access-token can generally be disregarded for Kubernetes. It would be used if the identity provider was managing roles and permissions for the users but that is done in Kubernetes itself with RBAC. The id-token is short lived while the refresh-token has longer expiration. The refresh-token is used to fetch a new id-token when the id-token expires.
{
"access_token":"ACCESS_TOKEN",
"refresh_token":"REFRESH_TOKEN",
"id_token": "ID_TOKEN",
"token_type":"bearer",
"scope": "openid groups profile email"
}
To introspect the ID_TOKEN token run:
$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/introspect \
-d token=${ID_TOKEN} \
--user kubernetes-auth:${OIDC_CLIENT_SECRET} | jq
The result will be like the following:
{
"exp": 1601323086,
"iat": 1601322186,
"aud": "kubernetes",
"typ": "ID",
"azp": "kubernetes",
"preferred_username": "alice",
"email_verified": false,
"acr": "1",
"groups": [
"capsule.clastix.io"
],
"client_id": "kubernetes",
"username": "alice",
"active": true
}
Configuring Kubernetes for OIDC Authentication requires adding several parameters to the API Server. Please, refer to the documentation for details and examples. Most likely, your kube-apiserver.yaml manifest will looks like the following:
The configuration file approach allows you to configure multiple JWT authenticators, each with a unique issuer.url and issuer.discoveryURL. The configuration file even allows you to specify CEL expressions to map claims to user attributes, and to validate claims and user information.
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://${OIDC_ISSUER}
audiences:
- kubernetes
- kubernetes-auth
audienceMatchPolicy: MatchAny
claimMappings:
username:
claim: 'email'
prefix: ""
groups:
claim: 'groups'
prefix: ""
certificateAuthority: <PEM encoded CA certificates>
This file must be present and consistent across all kube-apiserver instances in the cluster. Add the following flag to the kube-apiserver manifest:
spec:
containers:
- command:
- kube-apiserver
...
- --authentication-configuration-file=/etc/kubernetes/authentication/authentication.yaml
spec:
containers:
- command:
- kube-apiserver
...
- --oidc-issuer-url=https://${OIDC_ISSUER}
- --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
- --oidc-client-id=kubernetes
- --oidc-username-claim=preferred_username
- --oidc-groups-claim=groups
- --oidc-username-prefix=-
As reference, here is an example of a KinD configuration for OIDC Authentication, which can be useful for local testing:
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
extraArgs:
oidc-issuer-url: https://${OIDC_ISSUER}
oidc-username-claim: preferred_username
oidc-client-id: kubernetes
oidc-username-prefix: "keycloak:"
oidc-groups-claim: groups
oidc-groups-prefix: "keycloak:"
enable-admission-plugins: PodNodeSelector
There are two options to use kubectl with OIDC:
Plugin
One way to use OIDC authentication is the use of a kubectl plugin. The Kubelogin Plugin for kubectl simplifies the process of obtaining an OIDC token and configuring kubectl to use it. Follow the link to obtain installation instructions.
kubectl oidc-login setup \
--oidc-issuer-url=https://${OIDC_ISSUER} \
--oidc-client-id=kubernetes-auth \
--oidc-client-secret=${OIDC_CLIENT_SECRET}
Manual
To use the OIDC Authenticator, add an oidc user entry to your kubeconfig file:
$ kubectl config set-credentials oidc \
--auth-provider=oidc \
--auth-provider-arg=idp-issuer-url=https://${OIDC_ISSUER} \
--auth-provider-arg=idp-certificate-authority=/path/to/ca.crt \
--auth-provider-arg=client-id=kubernetes-auth \
--auth-provider-arg=client-secret=${OIDC_CLIENT_SECRET} \
--auth-provider-arg=refresh-token=${REFRESH_TOKEN} \
--auth-provider-arg=id-token=${ID_TOKEN} \
--auth-provider-arg=extra-scopes=groups
To use the --token option:
$ kubectl config set-credentials oidc --token=${ID_TOKEN}
Point the kubectl to the URL where the Kubernetes APIs Server is reachable:
$ kubectl config set-cluster mycluster \
--server=https://kube.projectcapsule.io:6443 \
--certificate-authority=~/.kube/ca.crt
If your APIs Server is reachable through the capsule-proxy, make sure to use the URL of the capsule-proxy.
Create a new context for the OIDC authenticated users:
$ kubectl config set-context alice-oidc@mycluster \
--cluster=mycluster \
--user=oidc
As user alice, you should be able to use kubectl to create some namespaces:
$ kubectl --context alice-oidc@mycluster create namespace wind-production
$ kubectl --context alice-oidc@mycluster create namespace wind-development
$ kubectl --context alice-oidc@mycluster create namespace water-marketing
Warning: once your ID_TOKEN expires, the kubectl OIDC Authenticator will attempt to refresh automatically your ID_TOKEN using the REFRESH_TOKEN. In case the OIDC uses a self signed CA certificate, make sure to specify it with the idp-certificate-authority option in your kubeconfig file, otherwise you’ll not able to refresh the tokens.
Velero is a backup and restore solution that performs data protection, disaster recovery and migrates Kubernetes cluster from on-premises to the Cloud or between different Clouds.
When coming to backup and restore in Kubernetes, we have two main requirements:
The first requirement aims to backup all the resources stored into etcd database, for example: namespaces, pods, services, deployments, etc. The second is about how to backup stateful application data as volumes.
The main limitation of Velero is the multi tenancy. Currently, Velero does not support multi tenancy meaning it can be only used from admin users and so it cannot provided “as a service” to the users. This means that the cluster admin needs to take care of users’ backup.
Assuming you have multiple tenants managed by Capsule, for example wind and water, as cluster admin, you can to take care of scheduling backups for:
Create a backup of the tenant solar. It consists in two different backups:
To back up the solar tenant selectively, label the tenant as:
kubectl label tenant wind capsule.clastix.io/tenant=solar
and create the backup
velero create backup solar-tenant \
--include-cluster-resources=true \
--include-resources=tenants.capsule.clastix.io \
--selector capsule.clastix.io/tenant=solar
resulting in the following Velero object:
apiVersion: velero.io/v1
kind: Backup
metadata:
name: solar-tenant
spec:
defaultVolumesToRestic: false
hooks: {}
includeClusterResources: true
includedNamespaces:
- '*'
includedResources:
- tenants.capsule.clastix.io
labelSelector:
matchLabels:
capsule.clastix.io/tenant: solar
metadata: {}
storageLocation: default
ttl: 720h0m0s
Create a backup of all the resources belonging to the solar tenant namespaces:
velero create backup solar-namespaces \
--include-cluster-resources=false \
--include-namespaces solar-production,solar-development,solar-marketing
resulting in the following Velero object:
apiVersion: velero.io/v1
kind: Backup
metadata:
name: solar-namespaces
spec:
defaultVolumesToRestic: false
hooks: {}
includeClusterResources: false
includedNamespaces:
- solar-production
- solar-development
- solar-marketing
metadata: {}
storageLocation: default
ttl: 720h0m0s
Velero requires an Object Storage backend where to store backups, you should take care of this requirement before using Velero.
To recover the tenant after a disaster, or to migrate it to another cluster, create a restore from the previous backups:
velero create restore --from-backup solar-tenant
velero create restore --from-backup solar-namespaces
Using Velero to restore a Capsule tenant can lead to an incomplete recovery of tenant because the namespaces restored with Velero do not have the OwnerReference field used to bind the namespaces to the tenant. For this reason, all restored namespaces are not bound to the tenant:
kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
water active 9 5 {"pool":"water"} 34m
wind active 9 8 {"pool":"wind"} 33m
solar active 9 0 # <<< {"pool":"solar"} 54m
To avoid this problem you can use the script velero-restore.sh located under the hack/ folder:
./velero-restore.sh --kubeconfig /path/to/your/kubeconfig --tenant "wind" restore
Running this command, we are going to patch the tenant’s namespaces manifests that are actually ownerReferences-less. Once the command has finished its run, you got the tenant back.
kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
water active 9 5 {"pool":"water"} 44m
solar active 9 8 {"pool":"wind"} 43m
wind active 9 3 # <<< {"pool":"solar"} 12s
The Capsule dashboard allows you to track the health and performance of Capsule manager and tenants, with particular attention to resources saturation, server responses, and latencies. Prometheus and Grafana are requirements for monitoring Capsule.
Capsule can export OpenTelemetry traces for admission webhook requests through the OTLP gRPC exporter. Enable tracing with manager.options.tracing.enabled=true and set manager.options.tracing.endpoint to your collector or tracing backend, for example tempo.monitoring-system.svc:4317. The exporter supports insecure transport for local clusters, TLS server name overrides, gzip compression, export timeouts, custom OTLP headers, and basic authentication. For production, prefer manager.options.tracing.basicAuth.existingSecret so credentials are loaded from a Kubernetes Secret instead of being rendered into controller arguments. Use manager.options.tracing.sampleRatio to control trace volume.
Instrumentation for Tenants.
The following Metrics are exposed and can be used for monitoring:
# HELP capsule_tenant_condition Provides per tenant condition status for each condition
# TYPE capsule_tenant_condition gauge
capsule_tenant_condition{condition="Cordoned",tenant="solar"} 0
capsule_tenant_condition{condition="Ready",tenant="solar"} 1
# HELP capsule_tenant_namespace_condition Provides per namespace within a tenant condition status for each condition
# TYPE capsule_tenant_namespace_condition gauge
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="earth",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="fire",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="foild",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="green",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="solar",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Cordoned",target_namespace="wind",tenant="solar"} 0
capsule_tenant_namespace_condition{condition="Ready",target_namespace="earth",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="fire",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="foild",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="green",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="solar",tenant="solar"} 1
capsule_tenant_namespace_condition{condition="Ready",target_namespace="wind",tenant="solar"} 1
# HELP capsule_tenant_namespace_count Total number of namespaces currently owned by the tenant
# TYPE capsule_tenant_namespace_count gauge
capsule_tenant_namespace_count{tenant="solar"} 6
# HELP capsule_tenant_namespace_relationship Mapping metric showing namespace to tenant relationships
# TYPE capsule_tenant_namespace_relationship gauge
capsule_tenant_namespace_relationship{target_namespace="earth",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="fire",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="soil",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="green",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="solar",tenant="solar"} 1
capsule_tenant_namespace_relationship{target_namespace="wind",tenant="solar"} 1
# HELP capsule_tenant_resource_limit Current resource limit for a given resource in a tenant
# TYPE capsule_tenant_resource_limit gauge
capsule_tenant_resource_limit{resource="limits.cpu",resourcequotaindex="0",tenant="solar"} 2
capsule_tenant_resource_limit{resource="limits.memory",resourcequotaindex="0",tenant="solar"} 2.147483648e+09
capsule_tenant_resource_limit{resource="pods",resourcequotaindex="1",tenant="solar"} 7
capsule_tenant_resource_limit{resource="requests.cpu",resourcequotaindex="0",tenant="solar"} 2
capsule_tenant_resource_limit{resource="requests.memory",resourcequotaindex="0",tenant="solar"} 2.147483648e+09
# HELP capsule_tenant_resource_usage Current resource usage for a given resource in a tenant
# TYPE capsule_tenant_resource_usage gauge
capsule_tenant_resource_usage{resource="limits.cpu",resourcequotaindex="0",tenant="solar"} 0
capsule_tenant_resource_usage{resource="limits.memory",resourcequotaindex="0",tenant="solar"} 0
capsule_tenant_resource_usage{resource="namespaces",resourcequotaindex="",tenant="solar"} 2
capsule_tenant_resource_usage{resource="pods",resourcequotaindex="1",tenant="solar"} 0
capsule_tenant_resource_usage{resource="requests.cpu",resourcequotaindex="0",tenant="solar"} 0
capsule_tenant_resource_usage{resource="requests.memory",resourcequotaindex="0",tenant="solar"} 0
You can gather more information based on the status of the tenants. These can be scrapped via Kube-State-Metrics CustomResourcesState Metrics. With these you have the possibility to create custom metrics based on the status of the tenants.
Here as an example with the kube-prometheus-stack chart, set the following values:
kube-state-metrics:
rbac:
extraRules:
- apiGroups: [ "capsule.clastix.io" ]
resources: ["tenants"]
verbs: [ "list", "watch" ]
customResourceState:
enabled: true
config:
spec:
resources:
- groupVersionKind:
group: capsule.clastix.io
kind: "Tenant"
version: "v1beta2"
labelsFromPath:
name: [metadata, name]
metrics:
- name: "tenant_size"
help: "Count of namespaces in the tenant"
each:
type: Gauge
gauge:
path: [status, size]
commonLabels:
custom_metric: "yes"
labelsFromPath:
capsule_tenant: [metadata, name]
kind: [ kind ]
- name: "tenant_state"
help: "The operational state of the Tenant"
each:
type: StateSet
stateSet:
labelName: state
path: [status, state]
list: [Active, Cordoned]
commonLabels:
custom_metric: "yes"
labelsFromPath:
capsule_tenant: [metadata, name]
kind: [ kind ]
- name: "tenant_namespaces_info"
help: "Namespaces of a Tenant"
each:
type: Info
info:
path: [status, namespaces]
labelsFromPath:
tenant_namespace: []
commonLabels:
custom_metric: "yes"
labelsFromPath:
capsule_tenant: [metadata, name]
kind: [ kind ]
This example creates three custom metrics:
tenant_size is a gauge that counts the number of namespaces in the tenant.tenant_state is a state set that shows the operational state of the tenant.tenant_namespaces_info is an info metric that shows the namespaces of the tenant.Velero is a backup and restore solution that performs data protection, disaster recovery and migrates Kubernetes cluster from on-premises to the Cloud or between different Clouds.
When coming to backup and restore in Kubernetes, we have two main requirements:
The first requirement aims to backup all the resources stored into etcd database, for example: namespaces, pods, services, deployments, etc. The second is about how to backup stateful application data as volumes.
The main limitation of Velero is the multi tenancy. Currently, Velero does not support multi tenancy meaning it can be only used from admin users and so it cannot provided “as a service” to the users. This means that the cluster admin needs to take care of users’ backup.
Assuming you have multiple tenants managed by Capsule, for example wind and water, as cluster admin, you can to take care of scheduling backups for:
Create a backup of the tenant solar. It consists in two different backups:
To back up the solar tenant selectively, label the tenant as:
kubectl label tenant wind capsule.clastix.io/tenant=solar
and create the backup
velero create backup solar-tenant \
--include-cluster-resources=true \
--include-resources=tenants.capsule.clastix.io \
--selector capsule.clastix.io/tenant=solar
resulting in the following Velero object:
apiVersion: velero.io/v1
kind: Backup
metadata:
name: solar-tenant
spec:
defaultVolumesToRestic: false
hooks: {}
includeClusterResources: true
includedNamespaces:
- '*'
includedResources:
- tenants.capsule.clastix.io
labelSelector:
matchLabels:
capsule.clastix.io/tenant: solar
metadata: {}
storageLocation: default
ttl: 720h0m0s
Create a backup of all the resources belonging to the solar tenant namespaces:
velero create backup solar-namespaces \
--include-cluster-resources=false \
--include-namespaces solar-production,solar-development,solar-marketing
resulting in the following Velero object:
apiVersion: velero.io/v1
kind: Backup
metadata:
name: solar-namespaces
spec:
defaultVolumesToRestic: false
hooks: {}
includeClusterResources: false
includedNamespaces:
- solar-production
- solar-development
- solar-marketing
metadata: {}
storageLocation: default
ttl: 720h0m0s
Velero requires an Object Storage backend where to store backups, you should take care of this requirement before using Velero.
To recover the tenant after a disaster, or to migrate it to another cluster, create a restore from the previous backups:
velero create restore --from-backup solar-tenant
velero create restore --from-backup solar-namespaces
Using Velero to restore a Capsule tenant can lead to an incomplete recovery of tenant because the namespaces restored with Velero do not have the OwnerReference field used to bind the namespaces to the tenant. For this reason, all restored namespaces are not bound to the tenant:
kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
water active 9 5 {"pool":"water"} 34m
wind active 9 8 {"pool":"wind"} 33m
solar active 9 0 # <<< {"pool":"solar"} 54m
To avoid this problem you can use the script velero-restore.sh located under the hack/ folder:
./velero-restore.sh --kubeconfig /path/to/your/kubeconfig --tenant "wind" restore
Running this command, we are going to patch the tenant’s namespaces manifests that are actually ownerReferences-less. Once the command has finished its run, you got the tenant back.
kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
water active 9 5 {"pool":"water"} 44m
solar active 9 8 {"pool":"wind"} 43m
wind active 9 3 # <<< {"pool":"solar"} 12s