Kubernetes Multi-tenancy

Understanding and implementing multi-tenant architectures in Kubernetes

Multi-tenancy in Kubernetes refers to the architecture where a single Kubernetes cluster is shared by multiple users, teams, or workloads, known as "tenants." Each tenant operates in isolation with its own set of resources, policies, and security boundaries, while still benefiting from the shared infrastructure.

This approach offers significant advantages in terms of resource efficiency, operational overhead reduction, and cost optimization. However, it also presents challenges related to security isolation, resource fairness, and management complexity that must be carefully addressed.

Multi-tenancy models in Kubernetes can range from simple namespace-based separation to complex architectures involving multiple layers of isolation through various Kubernetes constructs and third-party solutions.

Multi-tenancy Models

Namespace-based Isolation

Simplest approach: Using Kubernetes namespaces to separate workloads
Logical separation: Resources are isolated by namespace boundaries
RBAC integration: Role-based access control tied to namespaces
Resource quotas: Limit resource consumption per namespace
Network policies: Control traffic between namespaces
Admission control: Enforce policies at namespace level
Limited isolation: Shares the same control plane and node resources

Cluster-based Isolation

Stronger separation: Dedicated cluster per tenant
Complete isolation: Full control plane and resource separation
Resource overhead: Higher infrastructure and management costs
Operational complexity: Managing multiple clusters
Fleet management: Tools like Cluster API or Fleet for consistent management
Disconnected tenants: Limited cross-tenant interaction capabilities
Maximum security: Complete workload isolation

Hybrid Models

Node-based isolation: Dedicated nodes for specific tenants
Control plane sharing: Single control plane, separated worker nodes
Virtual clusters: Virtualized control planes with shared infrastructure
Hierarchical namespaces: Nested namespace structures
Custom resource boundaries: Using CRDs to define tenant boundaries
Flexible security models: Balance between isolation and resource sharing
Cost-efficiency: Better resource utilization than pure cluster isolation

Namespace-based Multi-tenancy

Namespaces provide a mechanism for isolating groups of resources within a single Kubernetes cluster. They are the foundation of simple multi-tenancy models.

Namespace Configuration

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    tenant: tenant-a
    environment: production
---
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-b
  labels:
    tenant: tenant-b
    environment: staging

Resource Quotas

Resource quotas limit the aggregate resource consumption within a namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    # Pod count limits
    pods: "50"
    
    # Compute resource limits
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    
    # Storage limits
    requests.storage: 500Gi
    persistentvolumeclaims: "20"
    
    # API object count limits
    services: "30"
    configmaps: "50"
    secrets: "50"
    
    # Specific storage class limits
    ssd.storageclass.storage.k8s.io/requests.storage: 200Gi

Limit Ranges

Limit ranges enforce default resource limits for containers within a namespace:

apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-a-limits
  namespace: tenant-a
spec:
  limits:
  - default:
      memory: 512Mi
      cpu: 500m
    defaultRequest:
      memory: 256Mi
      cpu: 100m
    min:
      memory: 128Mi
      cpu: 50m
    max:
      memory: 2Gi
      cpu: 2
    type: Container

RBAC for Multi-tenancy

Role-based access control (RBAC) is crucial for limiting tenant access to their own resources:

Tenant Admin Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tenant-admin
  namespace: tenant-a
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["batch"]
  resources: ["jobs", "cronjobs"]
  verbs: ["*"]
# Explicitly exclude access to secrets containing specific prefixes
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]
  resourceNames: ["tenant-a-*"]

Tenant Admin Binding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-a-admin-binding
  namespace: tenant-a
subjects:
- kind: User
  name: tenant-a-admin
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  name: tenant-a-admins
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tenant-admin
  apiGroup: rbac.authorization.k8s.io

Tenant Viewer Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tenant-viewer
  namespace: tenant-a
rules:
- apiGroups: ["", "extensions", "apps", "batch"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]

For more complex scenarios, ClusterRoles can provide cross-namespace permissions while still respecting tenant boundaries:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tenant-namespaces-admin
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch"]
  resourceNames: ["tenant-a", "tenant-a-dev", "tenant-a-test"]

Network Policies

Network policies are essential for controlling traffic between tenant namespaces:

Default Deny Policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: tenant-a
spec:
  podSelector: {}  # Applies to all pods in the namespace
  policyTypes:
  - Ingress
  - Egress

Allow Internal Communication

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}  # Allow from any pod in the same namespace

Allow Specific External Access

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-ingress
  namespace: tenant-a
spec:
  podSelector:
    matchLabels:
      app: api-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          tenant: tenant-b
      podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Advanced Policy Controls

Pod Security Standards

Kubernetes Pod Security Standards provide predefined security profiles:

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Open Policy Agent (OPA) and Gatekeeper

For more sophisticated policy enforcement, OPA Gatekeeper provides custom policy definitions:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: tenant-require-labels
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces:
      - tenant-a
      - tenant-b
  parameters:
    labels:
      - key: "tenant"
        allowedRegex: "tenant-[a-z]+"
      - key: "cost-center"

Admission Controllers

Admission controllers like PodNodeSelector can restrict pod placement:

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: "tenant=tenant-a"

Resource Isolation Techniques

Node Isolation

Dedicated nodes can be assigned to specific tenants using taints and tolerations:

# Taint the nodes
kubectl taint nodes node1 tenant=tenant-a:NoSchedule

# Pod with matching toleration
apiVersion: v1
kind: Pod
metadata:
  name: tenant-a-app
  namespace: tenant-a
spec:
  tolerations:
  - key: tenant
    operator: Equal
    value: tenant-a
    effect: NoSchedule
  nodeSelector:
    tenant: tenant-a

Priority Classes

Priority classes ensure critical tenant workloads receive appropriate scheduling priority:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-a-high-priority
value: 100000
globalDefault: false
description: "High priority class for Tenant A critical workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-a-low-priority
value: 10000
globalDefault: false
description: "Low priority class for Tenant A non-critical workloads"

Virtual Clusters

Virtual clusters provide stronger isolation while still using a single physical cluster:

# Using vcluster to create a virtual cluster
vcluster create tenant-a-vcluster -n tenant-a-host --expose

# Connect to the virtual cluster
vcluster connect tenant-a-vcluster -n tenant-a-host

Benefits of virtual clusters include:

Dedicated control plane components
Separate API server with its own authentication
Independent scheduling decisions
Isolation of CustomResourceDefinitions
Multi-version support across different virtual clusters

Hierarchical Namespace Controller (HNC)

HNC enables namespace hierarchies for better organization of multi-tenant environments:

apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchyConfiguration
metadata:
  name: hierarchy
  namespace: tenant-a
spec:
  parent: org-tenants

This creates a parent-child relationship between namespaces, allowing:

Policy inheritance
RBAC propagation
Resource propagation
Hierarchical name structure

Security Considerations

Security Layers for Multi-tenancy

Pod Security Context
securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true
Runtime Security
- Container sandboxing (gVisor, Kata Containers)
- Runtime protection tools (Falco, Aqua, Sysdig)
- Audit logging at namespace level
Secrets Management
- External secrets stores (Vault, AWS Secrets Manager)
- RBAC restrictions on Secret objects
- Encryption at rest for sensitive data
Supply Chain Security
- Image scanning per tenant
- Admission control for container image sources
- Software Bill of Materials (SBOM) requirements

Security Vulnerabilities to Address

Node-level Escape
- Container escape vulnerabilities
- Kernel exploits
- Privileged containers
- HostPath volumes
Resource Exhaustion
- CPU throttling attacks
- Memory pressure
- Storage flooding
- API request flooding
Network Attacks
- Lateral movement between namespaces
- Exfiltration via shared services
- DNS poisoning
- Service mesh tampering
Control Plane Vulnerabilities
- Authentication bypasses
- Authorization flaws
- API server exploits
- etcd unauthorized access

Multi-tenancy Tools and Projects

Several projects and tools enhance Kubernetes multi-tenancy capabilities:

Kiosk: Management dashboard and self-service namespace provisioning
Capsule: Multi-tenancy operator for namespace-as-a-service
vCluster: Virtual Kubernetes clusters running inside namespace objects
Loft: Multi-tenancy platform with virtual clusters and self-service
HNC: Hierarchical Namespace Controller for nested namespaces
Kyverno: Policy management alternative to OPA Gatekeeper
Crossplane: Multi-cluster abstractions and resource management
Kubeflow: Multi-tenant machine learning platform on Kubernetes

Implementation Patterns

SaaS Multi-tenancy Pattern

# Namespace for shared services
apiVersion: v1
kind: Namespace
metadata:
  name: saas-shared-services
---
# Namespace for tenant-specific workloads
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a-workloads
  labels:
    tenant-id: tenant-a
---
# Namespace for tenant data (stronger isolation)
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a-data
  labels:
    tenant-id: tenant-a
    data-sensitivity: high

Enterprise Team Pattern

# Department namespace (parent)
apiVersion: v1
kind: Namespace
metadata:
  name: finance-department
  labels:
    department: finance
---
# Team namespace (child)
apiVersion: v1
kind: Namespace
metadata:
  name: finance-accounting
  labels:
    department: finance
    team: accounting
---
# Project namespace (grandchild)
apiVersion: v1
kind: Namespace
metadata:
  name: finance-accounting-reporting
  labels:
    department: finance
    team: accounting
    project: reporting

Environment-based Pattern

# Development environment for tenant A
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a-dev
  labels:
    tenant: tenant-a
    environment: development
---
# Staging environment for tenant A
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a-staging
  labels:
    tenant: tenant-a
    environment: staging
---
# Production environment for tenant A
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a-prod
  labels:
    tenant: tenant-a
    environment: production

Monitoring and Observability

Multi-tenant clusters require special consideration for monitoring:

Tenant-specific Metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tenant-a-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      tenant: tenant-a
  namespaceSelector:
    matchLabels:
      tenant: tenant-a
  endpoints:
  - port: metrics
    interval: 15s

Access Controls for Metrics
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: tenant-metrics-viewer namespace: tenant-a rules: - apiGroups: ["monitoring.coreos.com"] resources: ["servicemonitors", "podmonitors"] verbs: ["get", "list", "watch"]
Resource Usage Dashboards
- Tenant-specific Grafana dashboards
- Cost allocation visualization
- Namespace resource utilization
- Quota consumption tracking
Tenant Logging
- Log filtering by namespace/tenant
- Access controls on log data
- Log retention policies per tenant
- Shared vs. dedicated log storage

Cost Management

Managing costs in multi-tenant clusters involves:

Namespace Resource Tracking
apiVersion: v1 kind: Namespace metadata: name: tenant-a labels: tenant: tenant-a cost-center: cc-12345 department: engineering
Resource Utilization Tools
- Kubecost for tenant-specific cost allocation
- Prometheus metrics for resource utilization
- Custom resource reporting using labels
Chargeback Models
- Consumption-based billing
- Resource reservation billing
- Hybrid models with baselines and burst

Challenges and Solutions

Common Multi-tenancy Challenges

Noisy Neighbor Problem
- Challenge: One tenant consumes excessive resources affecting others
- Solution: Implement resource quotas, limits, and quality-of-service classes
- Example: Use guaranteed QoS for critical workloads
  resources: requests: memory: "1Gi" cpu: "500m" limits: memory: "1Gi" cpu: "500m"
Tenant Isolation Breaches
- Challenge: Potential for cross-tenant data or access leakage
- Solution: Network policies, RBAC, and admission controls
- Example: Implement OPA policies to enforce strict separation
Operational Complexity
- Challenge: Managing many tenants increases administrative overhead
- Solution: Automation, self-service provisioning, and standardization
- Example: Use Crossplane or Terraform for tenant provisioning
Version Management
- Challenge: Different tenants may require different API versions
- Solution: Virtual clusters or multiple physical clusters
- Example: Use vCluster to provide tenant-specific Kubernetes versions
Performance Predictability
- Challenge: Ensuring consistent performance for all tenants
- Solution: Node anti-affinity, topology spread constraints
- Example: Spread tenant workloads across failure domains
  topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: tenant: tenant-a

Best Practices

Start with clear tenant boundaries
- Define isolation requirements upfront
- Document trust boundaries and security assumptions
- Create consistent naming conventions
Layer security controls
- Defense in depth approach
- Multiple isolation mechanisms
- Regular security audits
Standardize tenant onboarding
- Automated provisioning workflows
- Template-based namespace creation
- Default policies and quotas
Plan for scalability
- Consider performance impact of many tenants
- Understand API server load implications
- Test with realistic tenant counts
Implement proactive monitoring
- Tenant-aware alerting
- Resource utilization tracking
- Security anomaly detection

Case Study: Large-scale SaaS Platform

A real-world SaaS platform implemented multi-tenancy with:

Hybrid isolation model
- Shared control plane
- Dedicated nodes for premium tenants
- Namespace isolation for standard tenants

Hierarchical organization

saas-platform (root)
├── shared-services
│   ├── ingress-controllers
│   ├── monitoring
│   └── logging
├── tenant-standard (group)
│   ├── tenant-a
│   ├── tenant-b
│   └── tenant-c
└── tenant-premium (group)
    ├── tenant-x
    ├── tenant-y
    └── tenant-z

Multi-layered security
- Network policies at all levels
- OPA Gatekeeper constraints
- Service mesh with mTLS
- Pod Security Standards enforcement
Resource management
- Guaranteed QoS for premium tenants
- Burstable QoS for standard tenants
- HPA for dynamic scaling within quotas
- VPA for right-sizing container resources
Tenant-specific customization
- ConfigMaps for tenant configuration
- GitOps workflow for tenant changes
- Self-service portal for tenant admins

Future of Kubernetes Multi-tenancy

The Kubernetes multi-tenancy landscape continues to evolve with emerging trends:

Stronger isolation primitives
- Enhanced container isolation
- Improved namespace boundary enforcement
- Hardware-level security features
Multi-cluster federation
- Centralized management of tenant clusters
- Cross-cluster service discovery
- Unified policy enforcement
Serverless multi-tenancy
- Function-level tenant isolation
- Event-driven multi-tenant architectures
- Pay-per-use tenant resource allocation
Zero-trust architectures
- Identity-based security models
- Fine-grained authorization
- Continuous verification

Conclusion

Kubernetes multi-tenancy offers significant benefits in resource efficiency and operational consolidation, but requires careful planning and implementation. By using the right combination of namespace isolation, RBAC, network policies, resource controls, and additional tools, organizations can create secure and efficient multi-tenant environments.

The key to successful multi-tenancy is understanding your specific requirements for isolation, determining the appropriate model, and implementing defense-in-depth security practices. As Kubernetes continues to mature, its multi-tenancy capabilities will expand, enabling even more sophisticated shared-cluster architectures.

Edit this page

Advanced Scheduling & Affinity

Understanding Kubernetes advanced scheduling capabilities and affinity rules

Welcome to From Docker to Kubernetes - Your Free Learning Journey Begins

Introducing our open-source platform for mastering containerization and orchestration.

On this page

Introduction to Kubernetes Multi-tenancy
Multi-tenancy Models
Namespace-based Multi-tenancy
RBAC for Multi-tenancy
Network Policies
Advanced Policy Controls
Resource Isolation Techniques
- Node Isolation
- Priority Classes
Virtual Clusters
Hierarchical Namespace Controller (HNC)
Security Considerations
- Security Layers for Multi-tenancy
- Security Vulnerabilities to Address
Multi-tenancy Tools and Projects
Implementation Patterns
Monitoring and Observability
Cost Management
Challenges and Solutions
- Common Multi-tenancy Challenges
Best Practices
Case Study: Large-scale SaaS Platform
Future of Kubernetes Multi-tenancy
Conclusion

Star on GitHub Create Issues