ManifestWorkReplicaSet

What is `ManifestWorkReplicaSet`

ManifestWorkReplicaSet is an aggregator API that uses ManifestWork and Placement to automatically create ManifestWork resources for clusters selected by Placement. It simplifies multi-cluster workload distribution by eliminating the need to manually create individual ManifestWork resources for each target cluster.

Feature Enablement

ManifestWorkReplicaSet is an alpha feature. When you install the hub with the latest clusteradm init, it is enabled by default and the cluster-manager deploys the cluster-manager-work-controller on the hub cluster.

Note: In OCM v1.4.x and earlier, or when the hub is installed another way (for example, with an older clusteradm or the Helm chart), ManifestWorkReplicaSet may be disabled. See Enable ManifestWorkReplicaSet on OCM v1.4.x and earlier.

Verify that ManifestWorkReplicaSet is deployed:

$ oc get ClusterManager cluster-manager -o yaml

Confirm that the cluster-manager-work-controller deployment appears in the status under status.generations:

kind: ClusterManager
metadata:
  name: cluster-manager
spec:
   ...
status:
   ...
  generations:
    ...
  - group: apps
    lastGeneration: 2
    name: cluster-manager-work-webhook
    namespace: open-cluster-management-hub
    resource: deployments
    version: v1
  - group: apps
    lastGeneration: 1
    name: cluster-manager-work-controller
    namespace: open-cluster-management-hub
    resource: deployments
    version: v1

Enable ManifestWorkReplicaSet on OCM v1.4.x and earlier

Edit the cluster-manager CR:

$ oc edit ClusterManager cluster-manager

Set the ManifestWorkReplicaSet feature gate to Enable in the workConfiguration field:

kind: ClusterManager
metadata:
  name: cluster-manager
spec:
   ...
  workConfiguration:
    featureGates:
    - feature: ManifestWorkReplicaSet
      mode: Enable

Then verify that cluster-manager-work-controller appears under status.generations. See Feature Enablement.

Disable ManifestWorkReplicaSet on OCM v1.4.x and earlier

Edit the cluster-manager CR and set the ManifestWorkReplicaSet feature gate to Disable:

kind: ClusterManager
metadata:
  name: cluster-manager
spec:
   ...
  workConfiguration:
    featureGates:
    - feature: ManifestWorkReplicaSet
      mode: Disable

Verify the feature is disabled: cluster-manager-work-controller no longer appears under status.generations, and the deployment is removed from the open-cluster-management-hub namespace:

$ oc get deployment cluster-manager-work-controller -n open-cluster-management-hub
Error from server (NotFound): deployments.apps "cluster-manager-work-controller" not found

When the feature is disabled, existing ManifestWorkReplicaSet resources are no longer reconciled, but the ManifestWorkReplicaSet objects and the ManifestWork objects they created remain on the cluster.

Overview

Here’s a simple example to get started with ManifestWorkReplicaSet.

This example deploys a CronJob and Namespace to a group of clusters selected by Placement.

apiVersion: work.open-cluster-management.io/v1alpha1
kind: ManifestWorkReplicaSet
metadata:
  name: mwrset-cronjob
  namespace: ocm-ns
spec:
  placementRefs:
    - name: placement-rollout-all # Name of a created Placement
      rolloutStrategy:
        type: All
  manifestWorkTemplate:
    deleteOption:
      propagationPolicy: SelectivelyOrphan
      selectivelyOrphans:
        orphaningRules:
          - group: ''
            name: ocm-ns
            namespace: ''
            resource: Namespace
    manifestConfigs:
      - feedbackRules:
          - jsonPaths:
              - name: lastScheduleTime
                path: .status.lastScheduleTime
              - name: lastSuccessfulTime
                path: .status.lastSuccessfulTime
            type: JSONPaths
        resourceIdentifier:
          group: batch
          name: sync-cronjob
          namespace: ocm-ns
          resource: cronjobs
    workload:
      manifests:
        - kind: Namespace
          apiVersion: v1
          metadata:
            name: ocm-ns
        - kind: CronJob
          apiVersion: batch/v1
          metadata:
            name: sync-cronjob
            namespace: ocm-ns
          spec:
            schedule: '* * * * *'
            concurrencyPolicy: Allow
            suspend: false
            jobTemplate:
              spec:
                backoffLimit: 2
                template:
                  spec:
                    containers:
                      - name: hello
                        image: 'quay.io/prometheus/busybox:latest'
                        args:
                          - /bin/sh
                          - '-c'
                          - date; echo Hello from the Kubernetes cluster

The placementRefs field uses the Rollout Strategy API to control how ManifestWork resources are applied to the selected clusters.

In the example above, the placementRefs references placement-rollout-all with a rolloutStrategy of All, which means the workload will be deployed to all selected clusters simultaneously.

Rollout Strategy

ManifestWorkReplicaSet supports three rollout strategy types to control how workloads are distributed across clusters. For detailed rollout strategy documentation, see the Placement rollout strategy section.

Note: The placement reference must be in the same namespace as the ManifestWorkReplicaSet.

All: Deploy to all selected clusters simultaneously

  placementRefs:
    - name: placement-rollout-all # Name of a created Placement
      rolloutStrategy:
        type: All

Progressive: Gradual rollout with configurable parameters

  placementRefs:
    - name: placement-rollout-progressive # Name of a created Placement
      rolloutStrategy:
        type: Progressive
        progressive:
          minSuccessTime: 5m
          progressDeadline: 10m
          maxFailures: 5%
          mandatoryDecisionGroups:
          - groupName: "prod-canary-west"
          - groupName: "prod-canary-east"

ProgressivePerGroup: Progressive rollout per decision group

  placementRefs:
    - name: placement-rollout-progressive-per-group # Name of a created Placement
      rolloutStrategy:
        type: ProgressivePerGroup
        progressivePerGroup:
          progressDeadline: 10m
          maxFailures: 2

Rollout Mechanism

The ManifestWorkReplicaSet rollout process is based on the Applied, Progressing and Degraded conditions of each ManifestWork. Progressing and Degraded conditions are not built-in but must be defined using Custom CEL Expressions in the manifestConfigs section.

Condition Requirements

Applied condition (required): Indicates whether the ManifestWork has been successfully applied to the managed cluster by the work agent.
- This is a built-in condition that is automatically set by the work agent.
- The rollout controller checks this condition first to ensure the work has been properly applied before evaluating other conditions.
- If the Applied condition is missing, has Status=False/Unknown, or has a stale ObservedGeneration, it will never proceed to the next cluster.
- This ensures that rollout decisions are based on current status information, not stale data from previous generations.
Progressing condition (required): Tracks whether the ManifestWork is currently being applied to the cluster. The rollout controller uses this condition to determine if a cluster deployment is in progress or completed.
- For rollout strategies Progressive and ProgressivePerGroup, this is a required condition to determine if the rollout can proceed to the next cluster.
- When the Progressing condition is False (and Applied is ready), the deployment is considered complete and succeeded, and the rollout will continue to the next cluster based on minSuccessTime.
- If this condition is not defined or has a stale ObservedGeneration, the rollout will never proceed to the next cluster.
Degraded condition (optional): Indicates if the ManifestWork has failed or encountered issues.
- This is an optional condition used only to determine failure states.
- If the Degraded condition status is True, the rollout may stop or count towards maxFailures.
- If this condition is not defined, the workload is assumed to never be degraded.
- If this condition exists but has a stale ObservedGeneration, the rollout will wait for it to catch up to avoid using stale status information.

Rollout Status Determination:

The rollout controller determines the status of each ManifestWork based on the combination of Applied, Progressing and Degraded condition values:

Applied	Progressing	Degraded	Rollout Status	Description
`False`	Any	Any	ToApply	Work has not been applied or condition is stale
`True`	`True`	`True`	Failed	Work is progressing but degraded
`True`	`True`	`False` or not set	Progressing	Work is being applied and is healthy
`True`	`False`	`False` or not set	Succeeded	Work has been successfully applied
`True`	Unknown/Not set	Any	Progressing	Conservative fallback: treat as still progressing

Example: Progressive Rollout with Custom Conditions

This example demonstrates a progressive rollout with custom CEL expressions that define Progressing and Degraded conditions for a Deployment:

apiVersion: work.open-cluster-management.io/v1alpha1
kind: ManifestWorkReplicaSet
metadata:
  name: mwrset
  namespace: default
spec:
  placementRefs:
    - name: placement-test # Name of a created Placement
      rolloutStrategy:
        type: Progressive
        progressive:
          minSuccessTime: 1m
          progressDeadline: 10m
          maxFailures: 5%
          maxConcurrency: 1
  manifestWorkTemplate:
    workload:
      manifests:
        - apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: hello
            namespace: default
          spec:
            selector:
              matchLabels:
                app: hello
            template:
              metadata:
                labels:
                  app: hello
              spec:
                containers:
                  - name: hello
                    image: busybox
                    command:
                      ["sh", "-c", 'echo "Hello, Kubernetes!" && sleep 36000']
    manifestConfigs:
      - resourceIdentifier:
          group: apps
          resource: deployments
          namespace: default
          name: hello
        conditionRules:
          - condition: Progressing
            type: CEL
            celExpressions:
              - |
                !(
                  (object.metadata.generation == object.status.observedGeneration) &&
                  has(object.status.conditions) &&
                  object.status.conditions.exists(c, c.type == 'Progressing' && c.status == 'True' && c.reason == 'NewReplicaSetAvailable')
                )
            messageExpression: |
              result ? "Applying" : "Completed"
          - condition: Degraded
            type: CEL
            celExpressions:
              - |
                (object.metadata.generation == object.status.observedGeneration) &&
                (has(object.status.readyReplicas) && has(object.status.replicas) && object.status.readyReplicas < object.status.replicas)
            messageExpression: |
              result ? "Degraded" : "Healthy"

Status tracking

The PlacementSummary shows the number of manifestWorks applied to the placement’s clusters based on the placementRef’s rolloutStrategy and total number of clusters. The manifestWorkReplicaSet Summary aggregate the placementSummaries showing the total number of applied manifestWorks to all clusters.

The manifestWorkReplicaSet has three status conditions;

PlacementVerified verify the placementRefs status; not exist or empty cluster selection.
PlacementRolledOut verify the rollout strategy status; progressing or complete.
ManifestWorkApplied verify the created manifestWork status; applied, progressing, degraded or available.

The manifestWorkReplicaSet determines the ManifestWorkApplied condition status based on the resource state (applied or available) of each manifestWork.

Here is an example.

apiVersion: work.open-cluster-management.io/v1alpha1
kind: ManifestWorkReplicaSet
metadata:
  name: mwrset-cronjob
  namespace: ocm-ns
spec:
  placementRefs:
    - name: placement-rollout-all
      ...
    - name: placement-rollout-progressive
      ...
    - name: placement-rollout-progressive-per-group
      ...
  manifestWorkTemplate:
     ...
status:
 conditions:
   - lastTransitionTime: '2023-04-27T02:30:54Z'
     message: ''
     reason: AsExpected
     status: 'True'
     type: PlacementVerified
   - lastTransitionTime: '2023-04-27T02:30:54Z'
     message: ''
     reason: Progressing
     status: 'False'
     type: PlacementRolledOut
   - lastTransitionTime: '2023-04-27T02:30:54Z'
     message: ''
     reason: AsExpected
     status: 'True'
     type: ManifestworkApplied
 placementSummary:
 - name: placement-rollout-all
   availableDecisionGroups: 1 (10 / 10 clusters applied)
   summary:
     applied: 10
     available: 10
     progressing: 0
     degraded: 0
     total: 10
 - name: placement-rollout-progressive
   availableDecisionGroups: 3 (20 / 30 clusters applied)
   summary:
     applied: 20
     available: 20
     progressing: 0
     degraded: 0
     total: 20
 - name: placement-rollout-progressive-per-group
   availableDecisionGroups: 4 (15 / 20 clusters applied)
   summary:
     applied: 15
     available: 15
     progressing: 0
     degraded: 0
     total: 15
 summary:
   applied: 45
   available: 45
   progressing: 0
   degraded: 0
   total: 45