This is the multi-page printable view of this section. Click here to print.
This is general advice you should consider before making Kubernetes Distribution consideration. They are partly relevant for Multi-Tenancy with Capsule.
For a complete configuration combining the workload and networking policies, see the combined Tenant baseline.
User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider). The rationale is that other central platform components, such as ArgoCD, Grafana, Headlamp, or Harbor, should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions.
Capsule relies on native Kubernetes RBAC, so it’s important to consider how the Kubernetes API handles user authentication.
By default, Kubernetes clusters pull images directly from upstream registries like docker.io, quay.io, ghcr.io, or gcr.io. In production environments, this can lead to issues, especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled).
To ensure availability, performance, and control over container images, it’s essential to provide an on-premise OCI mirror.
This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., docker.io).
This way, all nodes automatically benefit from caching without requiring developers to change image URLs.
In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical. Instead, a centralized secrets management system should be established, such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project OpenBao (formerly the Vault community fork).
To integrate these external secret stores with Kubernetes, the External Secrets Operator (ESO) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing.
If no external secret store is available, there should at least be a secure way to store sensitive data in Git. In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case; called the sops-operator.
Use this complete Tenant as a starting point for application namespaces. It
combines the compatible baseline policies from Workloads
and Networking into one ordered
spec.rules list. All namespaces in this Tenant receive the same baseline.
| Area | Combined configuration |
|---|---|
| Resources | Default CPU, memory, and ephemeral-storage requests; remove CPU limits; match memory limits to requests; default ephemeral-storage limits. |
| Capacity | Share one illustrative quota across all Tenant namespaces. |
| QoS | Deny BestEffort after resource mutation. |
| Placement | Use the default scheduler, exclude control-plane nodes, deny control-plane and wildcard tolerations, and reject Pods created with nodeName. |
| Workload types | Deny DaemonSets. |
| Pod isolation | Set hostUsers: false and manage the namespace’s Pod Security labels at restricted. |
| Images | Allow the approved registry path and require Always pull policy. |
| Priority | Default to and allow only tenant-standard. |
| Services | Allow only ClusterIP and deny every external IP. |
| Namespace trust | Deny adding or changing company.com/system on Tenant namespaces. |
solar, solar-owner, and registry.example.com/solar with your
Tenant, owner, and approved registry path. Publish immutable image references
in that registry; the example’s registry matcher does not require a digest.tenant-standard PriorityClass
before admitting workloads that use this Tenant. The Tenant references the
class; it does not create it.hostUsers: false.
For incompatible workloads, define a separate, reviewed namespace profile.v1.36. Applications must supply security
contexts that satisfy restricted; these rules do not generate those fields.Explore the ordered rules below. Select a card to open its documentation, or view, copy, and download the complete Tenant YAML, including ownership and PriorityClass settings.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
# Default and allow the tenant-standard PriorityClass.
priorityClasses:
default: tenant-standard
allowed: [tenant-standard]
rules:
# One shared budget across all namespaces in this Tenant.
- quota:
- name: application-budget
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.memory: 40Gi
requests.ephemeral-storage: 100Gi
limits.ephemeral-storage: 200Gi
pods: "100"
services: "100"
# Apply placement and user-namespace settings on Pod creation.
- mutate:
- action: merge
workloads:
security:
hostUsers: false
placement:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
- key: node-role.kubernetes.io/master
operator: DoesNotExist
enforce:
action: allow
workloads:
targets: [pod]
placement:
schedulers:
- exact: [default-scheduler]
# Defaults for application containers, sidecars, and init containers.
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
ephemeral-storage:
policy: Default
value: 1Gi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
ephemeral-storage:
policy: Default
value: 2Gi
# Normalize limits when a workload supplies Pod-level resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Deny BestEffort Pods after resource mutation.
- enforce:
action: deny
workloads:
targets: [pod]
qosClasses: [BestEffort]
# Reject control-plane and wildcard tolerations.
- enforce:
action: deny
workloads:
targets: [pod]
placement:
tolerations:
- key:
exact:
- node-role.kubernetes.io/control-plane
- node-role.kubernetes.io/master
- key: {exp: '^$'}
operators: [Exists]
# Check direct assignment only at creation; allow scheduler-bound updates.
- enforce:
action: deny
conditions:
- name: direct-node-assignment
expression: >-
request.operation == 'CREATE' && has(object.spec.nodeName) && object.spec.nodeName != ''
workloads:
targets: [pod]
# Keep kind-level denials separate from workload property checks.
- enforce:
action: deny
workloads:
targets: [daemonset]
# Allow approved images with Always pull policy.
- enforce:
action: allow
workloads:
targets:
- pod/containers
- pod/initcontainers
- pod/ephemeralcontainers
- pod/volumes
registries:
- exp: '^registry\.example\.com/solar/.+$'
policy: [Always]
# Allow only ClusterIP Services.
- enforce:
action: allow
services:
types: [ClusterIP]
# Deny all Service external IPs.
- enforce:
action: deny
services:
externalIPs: {}
# Apply and reconcile Pod Security labels on the Namespace itself.
- enforce:
action: allow
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
pod-security.kubernetes.io/enforce:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/enforce-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/warn:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/warn-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/audit:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/audit-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
# Reserve this trusted-platform label for administrator-managed namespaces.
- enforce:
action: deny
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
company.com/system:
values:
- exp: ".*"
One shared budget across all namespaces in this Tenant.
- quota:
- name: application-budget
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.memory: 40Gi
requests.ephemeral-storage: 100Gi
limits.ephemeral-storage: 200Gi
pods: "100"
services: "100"
Apply placement and user-namespace settings on Pod creation.
- mutate:
- action: merge
workloads:
security:
hostUsers: false
placement:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
- key: node-role.kubernetes.io/master
operator: DoesNotExist
enforce:
action: allow
workloads:
targets: [pod]
placement:
schedulers:
- exact: [default-scheduler]
Defaults for application containers, sidecars, and init containers.
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
ephemeral-storage:
policy: Default
value: 1Gi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
ephemeral-storage:
policy: Default
value: 2Gi
Normalize limits when a workload supplies Pod-level resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
Deny BestEffort Pods after resource mutation.
- enforce:
action: deny
workloads:
targets: [pod]
qosClasses: [BestEffort]
Reject control-plane and wildcard tolerations.
- enforce:
action: deny
workloads:
targets: [pod]
placement:
tolerations:
- key:
exact:
- node-role.kubernetes.io/control-plane
- node-role.kubernetes.io/master
- key: {exp: '^$'}
operators: [Exists]
Check direct assignment only at creation; allow scheduler-bound updates.
- enforce:
action: deny
conditions:
- name: direct-node-assignment
expression: >-
request.operation == 'CREATE' && has(object.spec.nodeName) && object.spec.nodeName != ''
workloads:
targets: [pod]
Keep kind-level denials separate from workload property checks.
- enforce:
action: deny
workloads:
targets: [daemonset]
Allow approved images with Always pull policy.
- enforce:
action: allow
workloads:
targets:
- pod/containers
- pod/initcontainers
- pod/ephemeralcontainers
- pod/volumes
registries:
- exp: '^registry\.example\.com/solar/.+$'
policy: [Always]
Allow only ClusterIP Services.
- enforce:
action: allow
services:
types: [ClusterIP]
Deny all Service external IPs.
- enforce:
action: deny
services:
externalIPs: {}
Apply and reconcile Pod Security labels on the Namespace itself.
- enforce:
action: allow
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
pod-security.kubernetes.io/enforce:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/enforce-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/warn:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/warn-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/audit:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/audit-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
Reserve this trusted-platform label for administrator-managed namespaces.
- enforce:
action: deny
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
company.com/system:
values:
- exp: ".*"
Capsule applies mutations before validation. On Pod creation, the resource
rules supply missing requests and storage limits, remove CPU limits, and align
memory limits with requests. Explicit requests and storage limits survive the
Default policies. Placement mutation adds the control-plane exclusions and
sets hostUsers: false; enforcement then checks the resulting Pod.
The workload policies target Pods, including controller-created Pods. The DaemonSet rule separately rejects the controller itself. Add explicit controller targets if you also want resource or image checks on saved controller templates. Keep the DaemonSet denial and conditional direct-assignment denial separate: adding workload properties to a targets-only rule changes its meaning.
The quota rule generates solar-application-budget, a
GlobalResourceQuota shared
across this Tenant’s namespaces. It bounds aggregate consumption; the resource
defaults do not impose maximum requests or reserve node capacity by themselves.
Retain capacity for failover and system services as described in
capacity planning.
The restricted Pod Security labels are managed values. The platform-label
denial checks added or changed metadata; unchanged existing values are skipped
on updates. Remove any existing tenant-controlled company.com/system labels
before using them as a network trust boundary.
The memory-overcommit and ExternalName examples on the individual pages are deliberate exceptions. Add them only under reviewed namespace selectors, with the selecting labels under platform control. Review rule order and scope when composing further rules so a later decision does not unintentionally relax the baseline.
Apply the native NetworkPolicy GlobalTenantResource alongside this Tenant. It distributes default-deny, same-Tenant, DNS, and platform-ingress policies; NetworkPolicies are not embedded in this Tenant manifest. Review its CNI requirements and the permissions needed to prevent tenants from adding policies that widen access.
Use the additional admission policies
for requirements such as PDB validation, emptyDir.sizeLimit, debugging access,
and certificate issuer restrictions. Those examples require their respective
admission mechanisms and are not installed by this Tenant.
Configure authentication, registry mirrors, and secret management at platform level. Review existing workloads and Services before rollout; admission policies do not remove previously admitted resources.
Use namespace rules to establish workload defaults and validate the resulting
Pods. Every Capsule policy example below is a complete Tenant manifest with an
owner and spec.rules. Replace solar and solar-owner with your Tenant name
and owner identity. Each example is an independent configuration; to adopt
multiple policies, combine their spec.rules entries into one Tenant manifest.
Namespace selectors can give different namespaces different profiles; keep the
labels selecting mandatory profiles under platform control.
See the combined Tenant baseline for resource, placement, image, and networking policies together. Add the PDB policy below for namespace profiles that must support routine maintenance.
Resource settings are both a scheduling contract and a runtime safety boundary. Kubernetes schedules a Pod from its resource requests, while the kubelet and container runtime enforce its resource limits. Correct values help the scheduler place Pods safely, keep one workload from affecting its neighbours, and give autoscalers meaningful data.
CPU and memory need different treatment:
See Resource Management for Pods and Containers for the complete Kubernetes behavior.
Teams commonly use one of the following patterns:
BestEffort QoS. The scheduler
reserves no CPU or memory, and any memory use exceeds its memory request.
This is suitable only for disposable workloads.Burstable QoS. The scheduler accounts for the
workload and it can use spare capacity. Without a memory limit, however, a
leak can consume all memory available on the node.Burstable QoS and allows
controlled bursts. CPU can be throttled at its limit, and memory above the
request is more exposed during node pressure.Guaranteed QoS. It
reserves the full declared capacity and reduces memory-pressure exposure,
but prevents CPU bursting and can lower cluster utilization.Burstable QoS. It preserves CPU bursting while
reserving and bounding memory. This is the recommended baseline for general
workloads.A CPU limit can be useful when a workload needs a fixed CPU ceiling. Exclusive
CPU allocation under the CPU Manager static
policy
additionally requires eligible Guaranteed containers with integer CPU
requests. Guaranteed should not be the automatic target for every production
Pod.
For every application container, sidecar, and init container:
For example:
apiVersion: v1
kind: Pod
metadata:
name: api
spec:
containers:
- name: api
image: registry.example.com/solar/api:1.0.0
imagePullPolicy: Always
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
memory: 512Mi
The baseline can be visualized as two independent resource decisions that produce one Pod QoS class:
flowchart TB workload["Container resources"] cpu["CPU<br/>request: 250m<br/>limit: none"] memory["Memory<br/>request: 512Mi<br/>limit: 512Mi"] cpuResult["Scheduled CPU share<br/>Can use spare CPU"] memoryResult["512Mi scheduled<br/>Bounded at 512Mi"] qos["Pod QoS: Burstable"] workload --> cpu workload --> memory cpu --> cpuResult memory --> memoryResult cpuResult --> qos memoryResult --> qos
This gives the scheduler an honest view of the CPU needed under contention and
the maximum memory the container can consume. It also lets the container use
idle CPU above 250m without being throttled by an artificial ceiling.
Declare the memory request explicitly. If a limit is present without a request and no admission mechanism supplies a default, Kubernetes copies the limit to the request. Relying on that default makes the intended scheduling contract less obvious to readers and policy tools.
Treat a different memory request and limit as a deliberate overcommit policy, not as a default. It can improve density, but a container using more memory than its request is more vulnerable to eviction when its node is under memory pressure.
Apply this baseline with workload resource rules:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Normalize Pod-level limits too, when a workload supplies Pod resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
Replace solar-owner with your owner identity and size the defaults for your
workloads. Default preserves existing requests, Remove deletes CPU limits,
and MatchRequest sets memory limits to the corresponding requests. These
resource policies mutate Pod creation before enforcement; ordinary Pod updates
are validated without resource mutation. They also cover controller-created
Pods. Add explicit controller targets
if you want template validation and resource mutation when controllers are saved.
OOMKilled container needs investigation and usually a
corrected value; increasing the limit blindly can hide a memory leak.Kubernetes derives a Pod’s Quality of Service (QoS) class from CPU and memory settings. With container-level resources:
Guaranteed: every container has equal, non-zero CPU and memory requests
and limits.Burstable: some CPU or memory resources are declared, but the Pod does not
meet the Guaranteed criteria.BestEffort: no CPU or memory requests or limits are declared.Where supported, Pod-level resources also participate in classification. Ephemeral-storage requests and limits do not determine QoS.
The recommended baseline intentionally produces Burstable Pods because CPU
limits are absent. Equal memory requests and limits make memory reservations
explicit, but a QoS class does not guarantee protection from eviction. See
Eviction and Capacity Planning for the
resource-specific decisions, including disk pressure.
Capsule can allow, deny, or audit QoS
classes. A good tenant baseline
is to reject BestEffort Pods and permit the intentional Burstable pattern
described above.
For example, reject BestEffort without forcing all workloads into
Guaranteed:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod]
qosClasses: [BestEffort]
Resource mutations run before this check. When combined with the recommended resource baseline, its defaults supply requests before QoS is evaluated. This QoS-only example does not add requests or limits, ensure every container is sized, or provide a disk or inode reservation.
QoS describes how resources are declared. A
PriorityClass
describes how important one Pod is relative to another. These mechanisms are
independent: a Burstable Pod can have a higher priority than a Guaranteed
Pod.
A PriorityClass is cluster-scoped and maps a name to an integer. Higher-priority Pods are considered earlier in the scheduling queue. By default, a pending higher-priority Pod can also preempt lower-priority Pods when that would make it schedulable. The kubelet also considers priority during node-pressure eviction. Priority does not reserve capacity and does not give a running container more CPU time; requests and limits still define that behavior.
Define a small set of classes with clear purposes. For example, a standard tenant class can be non-preempting:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: tenant-standard
value: 1000
preemptionPolicy: Never
globalDefault: false
description: "Default priority for tenant application workloads"
Workload owners select an approved class on the Pod template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
priorityClassName: tenant-standard
containers:
- name: api
image: registry.example.com/solar/api:1.0.0
imagePullPolicy: Always
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
memory: 512Mi
Use elevated, preempting classes only for workloads whose unavailability would
prevent the platform or other applications from recovering. Do not use them to
compensate for undersized requests or insufficient cluster capacity. In a
multi-tenant cluster, restrict which PriorityClasses each tenant may select;
otherwise a tenant could displace another tenant’s Pods. Capsule can allow
classes and assign a tenant
default, and Kubernetes
ResourceQuota can limit consumption by PriorityClass.
Application owners should size their workloads, while the platform supplies safe fallbacks and validation:
LimitRange for simple namespace defaults and minimum or maximum
values. Do not configure a default CPU limit, because that would reverse the
recommended policy.ResourceQuota, Global Resource
Quotas, or Resource
Pools to control aggregate tenant
consumption. Quotas are capacity boundaries; they do not replace accurate
per-container sizing.Before deploying a workload, verify that every container has a CPU request, no
CPU limit, and equal memory request and limit; that the expected QoS class is
Burstable; and that any selected PriorityClass is approved for the tenant.
The terms overprovisioning and overcommit are sometimes used interchangeably, although they describe different capacity strategies:
The right approach depends on the resource. CPU contention delays work; memory contention can terminate Pods. They should therefore have different defaults.
Kubernetes places Pods according to CPU requests, so the total CPU requests on a node normally cannot exceed its allocatable CPU. With no CPU limits, however, the potential CPU demand of the running workloads can be much higher than the node’s capacity. This is intentional CPU overcommit: containers can use otherwise idle CPU, and their requests determine their relative share when several containers need CPU at the same time.
The recommended baseline supports CPU overcommit safely when requests remain honest. Do not lower requests merely to fit more Pods onto a node. An undersized request makes scheduling and capacity reports misleading, reduces the workload’s CPU share during contention, and distorts CPU-utilization-based Horizontal Pod Autoscaling.
There is no universal safe CPU overcommit ratio. Choose it from workload measurements and service objectives, and account for correlated events such as traffic peaks, rollouts, batch schedules, and node failures. Watch for:
Memory is overcommitted when the sum of memory requests fits on a node but the workloads can collectively use more memory than the node provides. A common way to create this condition is to set memory requests lower than memory limits. The scheduler considers the lower requests, while each container can grow toward its higher limit.
Our baseline avoids deliberate application memory overcommit by setting each memory request equal to its limit. Node services, runtime overhead, and failover still need their own capacity budget; this setting alone does not prevent node memory pressure.
Use memory overcommit only as an explicit exception for workloads that are well understood, restart-tolerant, and unlikely to peak together. Keep a hard memory limit, retain node headroom, and expect increased eviction risk whenever usage exceeds the request. Avoid memory overcommit for critical or stateful workloads, uncertain memory profiles, and applications with expensive recovery.
Monitor the exception using both scheduling and runtime signals:
For a measured exception, this complete Tenant includes the recommended
baseline followed by a rule selecting only batch namespaces. The last rule
replaces the earlier MatchRequest limit policy for regular and init container
memory in namespaces labeled example.com/capacity-profile: batch:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
cpu:
policy: Default
value: 250m
memory:
policy: Default
value: 512Mi
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Normalize Pod-level limits too, when a workload supplies Pod resources.
- enforce:
action: deny
workloads:
targets: [pod]
resources:
limits:
cpu:
policy: Remove
memory:
policy: MatchRequest
# Override container memory limits only in batch namespaces.
- namespaceSelector:
matchLabels:
example.com/capacity-profile: batch
enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
limits:
memory:
policy: Ratio
value: "2"
A missing memory limit becomes twice its positive request; an explicit limit
above that ratio is denied. Earlier request defaults and CPU policies remain
in effect. If workloads also use Pod-level memory resources, configure that
limit policy for the intended burst too. A later Preserve policy instead
clears earlier management and constraints for that same target, field, and
resource; it does not undo
Kubernetes defaults. See resource policy ordering.
Even well-sized workloads need room for failover, rolling updates, sudden scale-outs, and the delay before new nodes become ready. Define the required headroom from those recovery objectives rather than relying on capacity that happens to be idle.
Headroom can be provided by keeping additional nodes running or, when using Cluster Autoscaler, by scheduling low-priority placeholder Pods with resource requests. Real workloads preempt the placeholders and use the reserved space immediately. The displaced placeholder Pods become pending and can trigger a replacement node. The Cluster Autoscaler overprovisioning guide describes this pattern.
Placeholder Pods need a dedicated priority below application Pods, but not below
the autoscaler’s expendable-Pod cutoff if they should trigger scale-up. The
application class must permit preemption to displace them: tenant-standard
with preemptionPolicy: Never above cannot do that. Test the replacement and
scale-down behavior, and keep application state out of placeholders. A
PriorityClass only decides which Pods give way; it does not create capacity
by itself.
Review reservation ratios separately for each node pool and tenant. Aggregate cluster averages can hide a saturated pool or a single noisy tenant. Revisit the ratios after workload growth, node-type changes, autoscaler changes, or a capacity-related incident.
Plan for the resource that can run out. A CPU/memory QoS class is neither a capacity reservation for other resources nor a general eviction ranking.
| Situation | What happens | What to plan |
|---|---|---|
| CPU contention | Work slows; CPU limits can throttle. CPU saturation is not a kubelet node-pressure eviction signal. | Realistic CPU requests, latency targets, and scale-out capacity. |
| Container memory limit or node OOM | The kernel can kill a container; its restart policy determines recovery. This differs from a kubelet evicting the Pod. | Measured memory bounds, system reservations, and investigation of OOMKilled. |
| Node memory pressure | Kubelet may evict Pods to reclaim memory. | Memory requests versus actual usage, priorities, and node headroom. |
| Disk space pressure | Kubelet first attempts reclamation and can then evict Pods. CPU/memory QoS is not protection. | Storage requests, logs, writable layers, image caches, and filesystem free space. |
| Inode or PID pressure | Pods can be evicted even with spare memory or disk bytes. Priority matters; these resources have no Pod requests. | File counts, process counts, and per-node limits. |
| Local ephemeral-storage limit exceeded | With storage accounting enabled, the Pod can be evicted even without node-wide disk pressure. | Per-container storage limits and bounded scratch volumes. |
For threshold handling and reclamation, see node-pressure eviction. CPU and memory limit behavior is described under resource management; local storage accounting has its own requirements and limits.
For a node-wide OOM, Linux process selection also uses kubelet-provided OOM score adjustments influenced by QoS and memory requests. That is a different mechanism from pressure-eviction ranking. See the kubelet OOM policy.
Scheduler preemption is a separate decision that makes room for a pending
higher-priority Pod; it can happen without node pressure. preemptionPolicy: Never prevents a Pod from preempting others, not from being a victim. See
priority and preemption.
A drain normally uses the Eviction API, which checks PodDisruptionBudgets. Kubelet node-pressure eviction does not honor a PDB; hard thresholds can terminate immediately. A PDB limits some voluntary disruptions and creates no spare capacity. See disruption budgets and the PDB maintenance policy below.
For memory, eviction ranking considers usage above the request first, then lower Pod priority, then usage minus the request. Disk-byte ranking uses storage consumption and requests for the affected filesystem, not CPU/memory QoS. Filesystem layout determines which usage is counted. Inode/PID pressure cannot compare against a Pod request for those resources. See the Kubernetes eviction ordering.
Consider two Pods at the same priority on a node using one filesystem for local Pod storage:
| Pod | CPU/memory QoS | Ephemeral-storage request | Local storage use |
|---|---|---|---|
| A | Guaranteed | None | 5Gi |
| B | Burstable | 8Gi | 4Gi |
Under disk-byte pressure, A exceeds its storage request while B does not. With
valid usage statistics, A ranks before B despite being Guaranteed. Adding CPU
limits to B would not fix the full disk. Monitor the actual filesystem signal;
inode exhaustion is a different case from running out of bytes.
Likewise, under memory pressure, a high-priority Pod using more than its memory request can rank before a lower-priority Pod still within its request. Pods within requests can still be evicted if the node cannot recover, for example when system daemons exceed their reservations.
Requests for ephemeral-storage inform scheduling. Limits cover accounted
container logs, writable layers, and disk-backed emptyDir usage, with
Pod-level accounting rules. They are not an instantaneous filesystem quota.
Memory-backed emptyDir consumes memory instead. Persistent volumes need their
own storage planning. See local ephemeral
storage and
memory-backed volumes.
For workloads with local scratch data, add measured storage defaults:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [pod/containers, pod/initcontainers]
resources:
requests:
ephemeral-storage:
policy: Default
value: 1Gi
limits:
ephemeral-storage:
policy: Default
value: 2Gi
These are fallback values, not maximum allowed sizes: Default preserves
explicit settings. Use quota to bound namespace consumption, configure log
rotation, and size image-cache space and inodes separately. Ephemeral-storage
policies cannot target Pod-level spec.resources.
24Gi allocatable memory each have only 48Gi after one fails, before
failover or rollout demand. 60Gi of requests would not fit that scenario.Evicted, OOMKilled, and FailedScheduling. kubectl top
covers CPU/memory usage; it does not establish disk, inode, or PID headroom.A PodDisruptionBudget (PDB) protects application availability during voluntary evictions. An overly restrictive budget can also stop a shared node from being emptied, delaying node upgrades or replacement. One tenant’s Pod can therefore hold up maintenance affecting other tenants. PDBs also constrain Cluster Autoscaler scale-down, so a blocked eviction can leave otherwise removable nodes running.
Common causes and the corresponding Capsule policies are:
| Cause | Operational effect | Capsule policy |
|---|---|---|
maxUnavailable: 0, minAvailable: "100%", or an integer minimum equal to the replica count | No healthy replica can be voluntarily evicted | Require evictableReplicas.min: 1 |
| Scaling down makes a percentage budget round up to all replicas | A previously usable budget permits no disruption; for example, minAvailable: "75%" allows one at four replicas but none at three | Check replica changes on the controller and /scale against the same minimum |
| Multiple PDBs select a Pod | The Eviction API can reject the request with HTTP 500 | Set allowOverlap: false |
An unhealthy Running Pod is covered by IfHealthyBudget while its application is already below budget | An unhealthy Pod can hold up a drain | Allow only AlwaysAllow in unhealthyPodEvictionPolicies |
The Eviction API
checks the live budget. AlwaysAllow permits unhealthy Running Pods to leave
even when that budget is unsatisfied; healthy Pods still obey it. An omitted
PDB eviction policy means IfHealthyBudget. See the
Kubernetes unhealthy eviction policies.
For a namespace profile that must support routine maintenance, use this complete Tenant example. It checks Pods and Deployment/StatefulSet templates, including PDB writes and supported replica changes:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- namespaceSelector:
matchLabels:
example.com/profile: production
enforce:
action: allow
workloads:
targets:
- pod
- deployment
- statefulset
disruptionBudgets:
allowOverlap: false
evictableReplicas:
min: 1
unhealthyPodEvictionPolicies:
- AlwaysAllow
The rule applies only to namespaces labeled example.com/profile: production.
Keep that label under platform control; omit namespaceSelector to apply the
policy throughout the Tenant. PDB authors must explicitly set
unhealthyPodEvictionPolicy: AlwaysAllow; Capsule validates it without adding
the field. No PDB is required or created. For the complete field and target
reference, see PDB enforcement.
Choose disruption allowances with the application owner. min: 1 permits the
only replica of a singleton to be evicted when its PDB allows that, so it cannot
promise uninterrupted service. Run enough replicas and place them across the
failure domains you need to tolerate. Add evictableReplicas.max only when you
also need an upper bound on the per-controller configured allowance. Scaling
to zero is exempt from the replica-count check; this policy does not impose a
minimum workload size.
status.disruptionsAllowed. After an eviction, a replacement may remain Pending
because of capacity, affinity, taints, or volume placement. Further evictions can
then be blocked despite a compliant policy. Reserve headroom
and test recovery on the nodes that will remain available.Before maintenance, inspect PDB currentHealthy, desiredHealthy, and
disruptionsAllowed alongside Pod readiness and scheduling events. Review all
matching selectors when an eviction fails. A policy change does not repair
existing objects, and concurrent admissions across separate resources can race.
Coordinate an application or budget repair with its owner when necessary;
deleting a conflicting PDB remains possible.
PDBs cover voluntary API evictions. Kubelet node-pressure eviction does not honor them, including under disk pressure. Keep resource sizing and eviction planning alongside these policies; a PDB does not provide spare resources or protect a node from exhaustion.
Use workload mutations to establish scheduling restrictions and enforcement to reject incompatible requests. The effective namespace profile is applied to new Pods, including Pods created by workload controllers.
Keep control-plane node role labels and taints managed by the platform. For ordinary tenant applications, add required node affinity that excludes both the current and legacy control-plane labels:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- mutate:
- action: merge
workloads:
placement:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
- key: node-role.kubernetes.io/master
operator: DoesNotExist
enforce:
action: allow
workloads:
targets: [pod]
placement:
schedulers:
- exact: [default-scheduler]
- enforce:
action: deny
workloads:
targets: [pod]
placement:
tolerations:
- key:
exact:
- node-role.kubernetes.io/control-plane
- node-role.kubernetes.io/master
# An empty key with Exists tolerates every taint key.
- key: {exp: '^$'}
operators: [Exists]
- enforce:
action: deny
conditions:
- name: direct-node-assignment
expression: >-
request.operation == 'CREATE' && has(object.spec.nodeName) && object.spec.nodeName != ''
workloads:
targets: [pod]
Affinity merge combines the exclusions with existing required constraints;
it does not replace an application’s other node restrictions. The scheduler
allow-list keeps this example on the native scheduler. The last rule rejects
Pods created with nodeName, which would bypass scheduling, while allowing
normal updates after the scheduler binds a Pod. Tolerations permit scheduling
onto tainted nodes; they do not select nodes or prevent kubelet pressure
evictions. See taints and
tolerations.
These settings apply to newly admitted Pods, not existing ones. Keep node
label/taint writes and pods/binding permissions restricted to platform
components. If you use another approved scheduler or different control-plane
labels, adapt the profile to that setup.
Reserve DaemonSets for platform-managed agents when application tenants should not run one Pod on every eligible node:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
workloads:
targets: [daemonset]
With no workload properties, targets apply the action to the kind itself. This rejects DaemonSet creation and main-resource updates in the Tenant’s namespaces; existing objects are not removed. Other kinds are unaffected. Keep this rule separate from resource or image policies, because adding those properties changes targets into policy scopes. See workload targets.
For supported Linux node pools, mutate[].workloads.security.hostUsers: false
maps users inside a Pod to
different host IDs. Root inside the container therefore does not become host
root. Enable this only after verifying kernel, runtime, and volume filesystem
support; host namespaces and some volume configurations are incompatible. See
Kubernetes user namespaces.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- mutate:
- action: merge
workloads:
security:
hostUsers: false
This sets the value on Pod creation, even if the submitted value was true.
It does not change the container’s runAsUser or migrate existing Pods. Apply
it to namespaces whose workloads and nodes support user namespaces, and use
Pod Security Admission for the remaining security-context requirements.
Capsule does not configure node sysctls or runtime prerequisites.
Use Pod Security
Admission
to apply the privileged, baseline, or restricted standards. Start new
application profiles at restricted; use warning and audit modes to assess
existing workloads before enforcing a stricter profile.
Distribute the namespace labels through managed metadata
rules. Explicitly select Namespace;
wildcard kinds do not opt into Namespace metadata management. This example
pins the policy to v1.36; choose a policy version supported by your cluster
and review it during upgrades.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
pod-security.kubernetes.io/enforce:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/enforce-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/warn:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/warn-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
pod-security.kubernetes.io/audit:
managed: restricted
required: true
values:
- exact: [restricted]
pod-security.kubernetes.io/audit-version:
managed: v1.36
required: true
values:
- exact: [v1.36]
Capsule sets managed values during admission and reconciles them on existing
namespaces for this unconditional rule. A user’s conflicting label value is
overwritten. Kubernetes then checks admitted Pods against the selected
standard; changing namespace labels does not retroactively evict running Pods.
For different profiles, use namespace selectors whose profile labels are
controlled by the platform. Prefer narrow, reviewed exceptions over allowing
tenants to select privileged themselves.
Use an approved registry and immutable image references. When requiring
imagePullPolicy: Always, remember that the runtime can still reuse cached
layers; Always does not mean downloading every layer again. Private-image
credential verification also depends on the kubelet’s
KubeletEnsureSecretPulledImages support and verification policy. Review the
image pull and credential verification
behavior
for your Kubernetes version.
For a namespace profile that requires registry checks on each container start:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
workloads:
targets:
- pod/containers
- pod/initcontainers
- pod/ephemeralcontainers
- pod/volumes
registries:
- exp: '^registry\.example\.com/solar/.+$'
policy: [Always]
This permits only references in that registry path with Always pull policy;
it validates the submitted policy rather than changing it. Registry allow-listing
is separate from registry authentication and image signing. Budget image-cache
storage and registry availability as part of capacity planning.
For ordinary application namespaces, start with ClusterIP Services and expose
applications through a platform-managed Gateway or Ingress. Give namespaces
that need other Service types an explicit profile through
Service enforcement rules.
See the combined Tenant baseline for Service restrictions together with workload and namespace policies.
| Setting | Default policy for application namespaces | Reason |
|---|---|---|
ClusterIP | Allow, including headless Services. | Supports internal service discovery. NetworkPolicy still needs to control who can connect. |
NodePort | Deny. | Opens a port on node addresses, creating an exposure path outside the managed application entry point. |
LoadBalancer | Deny unless approved. | Can provision externally reachable infrastructure and normally allocates node ports too. |
ExternalName | Deny unless approved. | Lets a tenant alias a Service name to another DNS name; review destinations before enabling it. |
spec.externalIPs | Deny on every Service type. | Lets a Service claim traffic for an IP independently of its type. |
These are recommended rules to install, not Capsule’s automatic defaults. See the Kubernetes Service types for their networking behavior. The upstream external IP deprecation proposal explains the traffic-interception risk of user-selected external IPs.
This complete Tenant applies the baseline to all its namespaces:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
services:
types: [ClusterIP]
- enforce:
action: deny
services:
externalIPs: {}
Replace solar and solar-owner with your Tenant name and owner identity. The
type allow-list rejects NodePort, LoadBalancer, and ExternalName.
Under action: deny, externalIPs: {} rejects every supplied external IP,
including IPv4 and IPv6, while allowing Services without external IPs. This
separate check matters because a ClusterIP Service can also set
spec.externalIPs.
These checks run on Service creation and updates. They do not remove existing Services or close existing exposure when a rule is installed. Review existing Services during rollout; subsequent updates must satisfy the effective rules.
Keep the baseline for ordinary namespaces and add narrowly scoped exceptions for workloads that need them. Protect the namespace labels selecting an exception so tenant users cannot grant themselves a more permissive profile. See Namespace metadata rules.
For example, this complete Tenant retains the baseline and permits an
ExternalName pointing to database.example.com only in namespaces labeled
example.com/service-profile: external-database:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: allow
services:
types: [ClusterIP]
- enforce:
action: deny
services:
externalIPs: {}
- namespaceSelector:
matchLabels:
example.com/service-profile: external-database
enforce:
action: allow
services:
types: [ExternalName]
externalNames:
hostnames:
- exact: [database.example.com]
In selected namespaces, ClusterIP remains allowed and the approved
ExternalName is added. Other hostnames, NodePort, LoadBalancer, and all
external IPs remain denied. Non-selected namespaces retain the baseline.
Hostname constraints alone do not allow the ExternalName type; the exception
must include both the type and its hostname constraint.
The examples are alternative configurations of the same Tenant. Combine the
desired spec.rules entries into one manifest when adopting other policies.
Review rule order and scope: later matching rules
can override earlier decisions for the same Service value.
An ExternalName allow-list checks the submitted DNS name, not its resolved IP
addresses. It does not restrict connections made directly to other names or
IPs. Keep egress controls in place and consider who controls the approved DNS
name. See ExternalName Services.
Some workloads need a dedicated load balancer. Treat that as a separate namespace profile, with a reviewed provider configuration:
LoadBalancer only in
selected namespaces. Require the appropriate class or provider annotations
for an internal load balancer where needed. Use
metadata rules for annotations and
enforcement conditions to deny
incompatible Service fields. An allowed address range alone does not make a
load balancer private.NodePort Service type does not stop
a permitted LoadBalancer from allocating node ports. If the implementation
routes directly to Pods, require spec.allocateLoadBalancerNodePorts: false
and reject explicit spec.ports[].nodePort values. Changing that flag on an
existing Service does not release previously allocated ports; remove them
explicitly. Other implementations need node ports, so constrain access with
node firewalls and provider controls. See load balancer node-port
allocation.spec.loadBalancerSourceRanges and verify the provider enforces them; an
unsupported provider can ignore this field. Capsule’s
LoadBalancer CIDR checks
validate both spec.loadBalancerIP and spec.loadBalancerSourceRanges.
They do not require source ranges specifically when loadBalancerIP is
present, so use a condition-based deny rule if missing source ranges must be
rejected. See the Service API.Service admission controls configuration; it does not establish tenant network isolation. Apply NetworkPolicies with a supporting CNI and test the actual ingress and egress paths. Address translation can affect which source or destination a policy sees, particularly with load balancers. See NetworkPolicy behavior.
Also review these paths outside the Service type rules:
EndpointSlice, legacy Endpoints, and
services/status through RBAC. Endpoint writes can redirect Service traffic;
keep them with trusted controllers unless a workload explicitly needs that
capability. See RBAC guidance on traffic
redirection.hostNetwork and hostPort. Blocking NodePort Services does not
restrict these Pod settings.ClusterIP backend can still be deliberately published by
those controllers; see Ingress enforcement.Use GlobalTenantResource to distribute a common network profile across Tenant namespaces. Start with default-deny ingress and egress, then allow the flows applications need. The CNI must support and enforce native Kubernetes NetworkPolicies.
Native NetworkPolicies are additive: an additional allow policy can widen the effective access. Protecting the replicated baseline alone is therefore not a mandatory isolation boundary. Restrict tenant permissions to create additional NetworkPolicies when the platform must own that boundary, or use an administrator-enforced policy mechanism supported by your CNI. Both source egress and destination ingress must allow a connection. See the Kubernetes policy-combination rules.
This example creates four policies per selected namespace. Verify replication
readiness and test same-Tenant access, cross-Tenant denial, DNS, and platform
ingress before relying on the profile. Replication readiness confirms resource
delivery; it does not prove the CNI has applied the rules. Tune
resyncPeriod for the number of namespaces you manage.
This complete, cluster-scoped GlobalTenantResource creates four native
networking.k8s.io/v1 NetworkPolicies in every namespace of every selected
Tenant. It has no metadata.namespace. tenantSelector: {} selects all
Tenants; replace it with a label selector to roll out the profile gradually.
Use resources[].namespaceSelector when only some namespaces need this profile.
The replication identity must
have permission to manage NetworkPolicies in the destination namespaces.
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: tenant-network-baseline
spec:
scope: Namespace
tenantSelector: {}
resyncPeriod: 60s
resources:
- policy:
creation: Owner
protect: true
deletion: Remove
rawItems:
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-default-deny
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-allow-same-tenant
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
ingress:
- from:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: "{{tenant.name}}"
egress:
- to:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: "{{tenant.name}}"
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-allow-dns
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
# Both selectors apply to the same destination Pod.
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: capsule-allow-platform-ingress
spec:
podSelector: {}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels:
company.com/system: "true"
Capsule renders {{tenant.name}} separately for each destination Tenant using
Fast Templates, and sets
the destination namespace on each generated policy. The selector uses Capsule’s
namespace membership label, capsule.clastix.io/tenant.
The resulting profile permits:
| Traffic | Allowance in this profile |
|---|---|
| Between Pods in the same Tenant, including the same namespace | Ingress and egress on all ports. |
| From Tenant Pods to cluster DNS | TCP and UDP port 53 to the selected DNS Pods. |
| From Pods in trusted platform namespaces to Tenant Pods | Ingress on all ports; narrow the Pod selectors and ports for your monitoring or ingress controllers. |
| To another Tenant, the internet, or other platform destinations | No allowance from this profile; add a reviewed policy for required flows. |
For namespace-level isolation within a Tenant, replace the same-tenant peers
with podSelector: {} so they select Pods only in the policy’s namespace.
Install the namespace metadata restriction below
on every application Tenant before trusting company.com/system.
Check your DNS deployment’s namespace and labels; the example assumes
kube-system and k8s-app: kube-dns. See DNS
troubleshooting.
NodeLocal DNSCache or host-networked DNS may need different CNI-specific
allowances. Test DNS over both UDP and TCP instead of relying on a fixed Service
IP. The platform workloads also need an egress policy permitting their
connections if their own namespaces isolate egress.
protect: true protects the replicated objects from tenant modification or
deletion through Capsule admission. It does not stop someone from creating
another NetworkPolicy. With deletion: Remove, removing the replication or a
destination from its selection removes the policies it created; plan those
changes as changes to network access.
The platform-ingress policy trusts namespaces carrying
company.com/system: "true". Reserve that label for administrator-managed
platform namespaces outside application Tenants. Restrict who can label those
namespaces through RBAC.
Use a metadata enforcement rule
to deny the label on application Tenant namespaces. This complete Tenant
replaces the legacy namespaceOptions.forbiddenLabels configuration:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: solar-owner
rules:
- enforce:
action: deny
metadata:
- apiGroups: [v1]
kinds: [Namespace]
labels:
company.com/system:
values:
- exp: ".*"
The explicit Namespace kind targets the Namespace object itself. The rule
rejects the label on creation and when an update adds or changes its value,
including "true", "false", and an empty value. Namespaces without the label
and updates removing it remain allowed. Apply this rule to every application
Tenant and merge it with the Service rules above when using both profiles.
Keep it unconditional so changing a namespace label cannot opt out of the
restriction. Review later rules that could override this denial.
Metadata validation skips unchanged values on updates. This rule therefore
does not remove an existing label or reject an update that leaves its value
unchanged. Remove any existing tenant-controlled company.com/system labels
before enabling the platform-ingress allowance.
As Capsule we try to provide a secure multi-tenant environment out of the box, there are however some additional Admission Policies you should consider to enforce best practices in your cluster. Since Capsule only covers the core multi-tenancy features, such as Namespaces, Resource Quotas, Network Policies, and Container Registries, Classes, you should consider using an additional Admission Controller to enforce best practices on workloads and other resources.
Create custom Policies and reuse data provided via Tenant Status to enforce your own rules.
Let’s say we have the following namespaced ObjectBucketClaim resource:
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: admission-class
namespace: solar-production
finalizers:
- objectbucket.io/finalizer
labels:
bucket-provisioner: openshift-storage.ceph.rook.io-bucket
spec:
additionalConfig:
maxSize: 2G
bucketName: test-some-uid
generateBucketName: test
objectBucketName: obc-test-test
storageClassName: ocs-storagecluster-ceph-rgw
However since we are allowing Tenant Users to create these ObjectBucketClaims we might want to consider validating the storageClassName field to ensure that only allowed StorageClasses are used.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-tenant-class
spec:
validationFailureAction: Enforce
rules:
- name: restrict-storage-class
context:
- name: classes
apiCall:
urlPath: "/apis/capsule.clastix.io/v1beta2/tenants"
jmesPath: "items[?contains(status.namespaces, '{{ request.namespace }}')].status.classes | [0]"
- name: storageClass
variable:
jmesPath: "request.object.spec.storageClassName || 'NONE'"
match:
resources:
kinds:
- ObjectBucketClaim
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "storageclass {{ storageClass }} is not allowed in tenant ({{classes.storage}})"
deny:
conditions:
- key: "{{classes.storage}}"
operator: AnyNotIn
value: "{{ storageClass }}"Policies to harden workloads running in a multi-tenant environment.
Pod Disruption Budgets (PDBs) are a way to limit the number of concurrent disruptions to your Pods. In multi-tenant environments, it is recommended to enforce the usage of PDBs to ensure that tenants do not accidentally or maliciously block cluster operations.
A PodDisruptionBudget which sets its maxUnavailable value to zero prevents all voluntary evictions including Node drains which may impact maintenance tasks. This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field it must be greater than zero.
---
# Source: https://kyverno.io/policies/other/pdb-maxunavailable/pdb-maxunavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: pdb-maxunavailable
annotations:
policies.kyverno.io/title: PodDisruptionBudget maxUnavailable Non-Zero
policies.kyverno.io/category: Other
kyverno.io/kyverno-version: 1.9.0
kyverno.io/kubernetes-version: "1.24"
policies.kyverno.io/subject: PodDisruptionBudget
policies.kyverno.io/description: >-
A PodDisruptionBudget which sets its maxUnavailable value to zero prevents
all voluntary evictions including Node drains which may impact maintenance tasks.
This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field
it must be greater than zero.
spec:
validationFailureAction: Enforce
background: false
rules:
- name: pdb-maxunavailable
match:
any:
- resources:
kinds:
- PodDisruptionBudget
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "The value of maxUnavailable must be greater than zero."
pattern:
spec:
=(maxUnavailable): ">0"apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: pdb-maxunavailable
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["policy"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["poddisruptionbudgets"]
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validations:
- expression: |
!has(object.spec.maxUnavailable) ||
string(object.spec.maxUnavailable).contains('%') ||
object.spec.maxUnavailable > 0
message: "The value of maxUnavailable must be greater than zero or a percentage."
reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: pdb-maxunavailable-binding
spec:
policyName: pdb-maxunavailable
validationActions: ["Deny"]When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget, if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget it may prevent voluntary disruptions including Node drains which may impact routine maintenance tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have a matching PodDisruptionBudget to ensure these two values do not match.
---
# Source: https://kyverno.io/policies/other/pdb-minavailable/pdb-minavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: pdb-minavailable-check
annotations:
policies.kyverno.io/title: Check PodDisruptionBudget minAvailable
policies.kyverno.io/category: Other
kyverno.io/kyverno-version: 1.9.0
kyverno.io/kubernetes-version: "1.24"
policies.kyverno.io/subject: PodDisruptionBudget, Deployment, StatefulSet
policies.kyverno.io/description: >-
When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
it may prevent voluntary disruptions including Node drains which may impact routine maintenance
tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have
a matching PodDisruptionBudget to ensure these two values do not match.
spec:
validationFailureAction: Enforce
background: false
rules:
- name: pdb-minavailable
match:
any:
- resources:
kinds:
- Deployment
- StatefulSet
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
preconditions:
all:
- key: "{{`{{ request.operation | 'BACKGROUND' }}`}}"
operator: AnyIn
value:
- CREATE
- UPDATE
- key: "{{`{{ request.object.spec.replicas | '1' }}`}}"
operator: GreaterThan
value: 0
context:
- name: minavailable
apiCall:
urlPath: "/apis/policy/v1/namespaces/{{`{{ request.namespace }}`}}/poddisruptionbudgets"
jmesPath: "items[?label_match(spec.selector.matchLabels, `{{`{{ request.object.spec.template.metadata.labels }}`}}`)] | [0].spec.minAvailable | default(`0`)"
validate:
message: >-
The matching PodDisruptionBudget for this resource has its minAvailable value equal to the replica count
which is not permitted.
deny:
conditions:
any:
- key: "{{`{{ request.object.spec.replicas }}`}}"
operator: Equals
value: "{{`{{ minavailable }}`}}"PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.Introducing a PDB where there are already matching Pod controllers may pose a problem if the author is unaware of the existing replica count. This policy ensures that the minAvailable value is not greater or equal to the replica count of any matching existing Deployment. If other Pod controllers should also be included in this check, additional rules may be added to the policy which match those controllers.
---
# Source: https://kyverno.io/policies/other/deployment-replicas-higher-than-pdb/deployment-replicas-higher-than-pdb/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: deployment-replicas-higher-than-pdb
annotations:
policies.kyverno.io/title: Ensure Deployment Replicas Higher Than PodDisruptionBudget
policies.kyverno.io/category: Other
policies.kyverno.io/subject: PodDisruptionBudget, Deployment
kyverno.io/kyverno-version: 1.11.4
kyverno.io/kubernetes-version: "1.27"
policies.kyverno.io/description: >-
PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.
Introducing a PDB where there are already matching Pod controllers may pose a problem if the author
is unaware of the existing replica count. This policy ensures that the minAvailable value is not
greater or equal to the replica count of any matching existing Deployment. If other Pod controllers
should also be included in this check, additional rules may be added to the policy which match those
controllers.
spec:
validationFailureAction: Enforce
background: true
rules:
- name: deployment-replicas-greater-minAvailable
match:
any:
- resources:
kinds:
- PodDisruptionBudget
operations:
- CREATE
- UPDATE
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
context:
- name: deploymentreplicas
apiCall:
jmesPath: items[?label_match(`{{`{{ request.object.spec.selector.matchLabels }}`}}`, spec.template.metadata.labels)] || `[]`
urlPath: /apis/apps/v1/namespaces/{{`{{request.namespace}}`}}/deployments
preconditions:
all:
- key: '{{`{{ length(deploymentreplicas) }}`}}'
operator: GreaterThan
value: 0
- key: '{{`{{ request.object.spec.minAvailable || "" }}`}}'
operator: NotEquals
value: ''
validate:
message: >-
PodDisruption budget minAvailable ({{`{{ request.object.spec.minAvailable }}`}}) cannot be
greater than or equal to the replica count of any matching existing Deployment.
There are {{`{{ length(deploymentreplicas) }}`}} Deployments which match this labelSelector
having {{`{{ deploymentreplicas[*].spec.replicas }}`}} replicas.
foreach:
- list: deploymentreplicas
deny:
conditions:
all:
- key: "{{`{{ request.object.spec.minAvailable }}`}}"
operator: GreaterThanOrEquals
value: "{{`{{ element.spec.replicas }}`}}"When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget, if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget it may prevent voluntary disruptions including Node drains which may impact routine maintenance tasks and disrupt operations. This policy checks incoming CNPG Clusters and their .spec.enablePDB setting.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: pdb-cnpg-cluster-validation
annotations:
policies.kyverno.io/title: Check PodDisruptionBudget minAvailable for cnpgCluster
policies.kyverno.io/category: Other
kyverno.io/kyverno-version: 1.9.0
kyverno.io/kubernetes-version: "1.24"
policies.kyverno.io/subject: PodDisruptionBudget, Cluster
policies.kyverno.io/description: >-
When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
it may prevent voluntary disruptions including Node drains which may impact routine maintenance
tasks and disrupt operations. This policy checks incoming CNPG Clusters and their .spec.enablePDB setting.
spec:
validationFailureAction: Enforce
background: false
rules:
- name: pdb-cnpg-cluster-validation
match:
any:
- resources:
kinds:
- postgresql.cnpg.io/v1/Cluster
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
preconditions:
any:
- key: "{{request.operation || 'BACKGROUND'}}"
operator: AnyIn
value:
- CREATE
- UPDATE
validate:
message: >-
Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2.
deny:
conditions:
all:
- key: "{{request.object.spec.enablePDB }}"
operator: Equals
value: true
- key: "{{request.object.spec.instances }}"
operator: LessThan
value: 2apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: pdb-cnpg-cluster-validation
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["postgresql.cnpg.io"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["clusters"]
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validations:
- expression: |
!has(object.spec.enablePDB) ||
object.spec.enablePDB == false ||
(has(object.spec.instances) && object.spec.instances >= 2)
message: "Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2."
messageExpression: |
'Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2. Current instances: ' +
string(has(object.spec.instances) ? object.spec.instances : 1)
reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: pdb-cnpg-cluster-validation-binding
spec:
policyName: pdb-cnpg-cluster-validation
validationActions: ["Deny"]By Defaults emptyDir Volumes do not have any limits. This could lead to a situation, where a tenant fills up the node disk. To avoid this, you can enforce limits on emptyDir volumes. You may also consider restricting the usage of emptyDir with the medium: Memory option, as this could lead to memory exhaustion on the node.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tenant-workload-restrictions
spec:
rules:
- name: default-emptydir-sizelimit
match:
any:
- resources:
kinds:
- Pod
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
mutate:
foreach:
- list: "request.object.spec.volumes[]"
preconditions:
all:
- key: "{{element.keys(@)}}"
operator: AnyIn
value: emptyDir
- key: "{{element.emptyDir.sizeLimit || ''}}"
operator: Equals
value: ''
patchesJson6902: |-
- path: "/spec/volumes/{{elementIndex}}/emptyDir/sizeLimit"
op: add
value: 250Mi Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the kubectl debug functionality and attach a temporary container to an existing Pod. This may potentially be used to gain access to unauthorized information executing inside one or more containers in that Pod. This policy blocks the use of ephemeral containers.
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: block-ephemeral-containers
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
# 1) Regular Pods (ensure spec doesn't carry ephemeralContainers)
- apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
operations: ["CREATE","UPDATE"]
scope: "Namespaced"
# 2) Subresource used by `kubectl debug` to inject ephemeral containers
- apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods/ephemeralcontainers"]
operations: ["UPDATE","CREATE"] # UPDATE is typical, CREATE included for future-proofing
scope: "Namespaced"
validations:
# Deny any request that targets the pods/ephemeralcontainers subresource
- expression: request.subResource != "ephemeralcontainers"
message: "Ephemeral (debug) containers are not permitted (subresource)."
# For direct Pod create/update, allow only if the field is absent or empty
- expression: >
!has(object.spec.ephemeralContainers) ||
size(object.spec.ephemeralContainers) == 0
message: "Ephemeral (debug) containers are not permitted in Pod specs."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: block-ephemeral-containers-binding
spec:
policyName: block-ephemeral-containers
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists# Source: https://kyverno.io/policies/other/block-ephemeral-containers/block-ephemeral-containers/
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: block-ephemeral-containers
annotations:
policies.kyverno.io/title: Block Ephemeral Containers
policies.kyverno.io/category: Other
policies.kyverno.io/severity: medium
kyverno.io/kyverno-version: 1.6.0
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kubernetes-version: "1.23"
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the
`kubectl debug` functionality and attach a temporary container to an existing Pod.
This may potentially be used to gain access to unauthorized information executing inside
one or more containers in that Pod. This policy blocks the use of ephemeral containers.
spec:
validationFailureAction: Enforce
background: true
rules:
- name: block-ephemeral-containers
match:
any:
- resources:
kinds:
- Pod
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "Ephemeral (debug) containers are not permitted."
pattern:
spec:
X(ephemeralContainers): "null"Often when working in multi-tenant environments, you want to ensure that tenants are not using ClusterIssuers to issue certificates, but rather use namespaced Issuers within their own namespace. This policy enforces that cert-manager.io/v1/Certificate resources do not reference ClusterIssuers and that the Issuer referenced is in the same namespace as the Certificate.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: certificates-only-local-issuer
spec:
validationFailureAction: Enforce
background: true
rules:
- name: deny-clusterissuer-in-certificates
match:
any:
- resources:
kinds:
- cert-manager.io/v1/Certificate
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "Certificates must not reference ClusterIssuers; use a namespaced Issuer in the same namespace."
deny:
conditions:
any:
- key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
operator: Equals
value: "ClusterIssuer"
- name: deny-cross-namespace-issuerref-in-certificates
match:
any:
- resources:
kinds:
- cert-manager.io/v1/Certificate
validate:
message: "Certificates must reference an Issuer in the same namespace (spec.issuerRef.namespace must be empty or equal to the Certificate namespace)."
deny:
conditions:
any:
# If issuerRef.namespace is set and differs from the Certificate namespace -> deny
- key: "{{request.object.spec.issuerRef.namespace || '' }}"
operator: NotEquals
value: ""
# AND also not equal to request namespace
- key: "{{ request.object.spec.issuerRef.namespace || request.namespace }}"
operator: NotEquals
value: "{{ request.namespace }}"Deny to usage of ClusterIssuers in Gateways by checking for the cert-manager.io/cluster-issuer annotation. This ensures that tenants use namespaced issuer mechanisms instead.
This requires extra permissions to allow Kyverno to read Gateway resources:
admissionController:
rbac:
clusterRole:
extraResources:
- apiGroups: ["gateway.networking.k8s.io"]
resources: ["*"]
verbs: ["get", "list", "watch"]
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: gateways-deny-cluster-issuer-annotation
spec:
validationFailureAction: Enforce
background: false
rules:
- name: deny-cert-manager-cluster-issuer-annotation
match:
any:
- resources:
kinds:
- gateway.networking.k8s.io/v1/Gateway
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
validate:
message: "Gateways must not use cert-manager.io/cluster-issuer; use namespaced issuer mechanisms instead."
deny:
conditions:
any:
- key: "{{ request.object.metadata.annotations.\"cert-manager.io/cluster-issuer\" || '' }}"
operator: NotEquals
value: ""Allow certain ClusterIssuers within Tenants:
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: certificates-restrict-clusterissuer-by-tenant-label
spec:
validationFailureAction: Enforce
background: true
rules:
- name: allow-clusterissuer-only-when-managed-by-matches
match:
any:
- resources:
kinds:
- cert-manager.io/v1/Certificate
namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: Exists
context:
- name: certManagedBy
variable:
jmesPath: request.object.metadata.labels."capsule.clastix.io/managed-by" || ''
- name: clusterIssuerManagedBy
apiCall:
urlPath: /apis/cert-manager.io/v1/clusterissuers/{{ request.object.spec.issuerRef.name }}
jmesPath: metadata.labels."company.com/tenant" || ''
default: ""
preconditions:
all:
- key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
operator: Equals
value: ClusterIssuer
- key: "{{request.operation || 'BACKGROUND'}}"
operator: AnyIn
value:
- CREATE
- UPDATE
validate:
message: >-
ClusterIssuer is only allowed when the Certificate label
capsule.clastix.io/managed-by ({{certManagedBy}}) matches the referenced ClusterIssuer label
company.com/tenant ({{clusterIssuerManagedBy}}).
deny:
conditions:
any:
- key: "{{certManagedBy}}"
operator: Equals
value: ""
- key: "{{clusterIssuerManagedBy}}"
operator: Equals
value: ""
- key: "{{certManagedBy}}"
operator: NotEquals
value: "{{clusterIssuerManagedBy}}"