MD.Status.ReadyReplicas changes from 3 to 0 when machineset_controller updateStatus() hits "Unable to retrieve Node status" error

[jessehu]

What steps did you take and what happened?

When calling clusterctlclient.Client.ApplyUpgrade(upgrade) to upgrade CAPI core components (its version is not changed) and a CAPI Infra Provider component(the version is changed), there is a very low probability that capi-controller-manager pod is restarted. Both capi-controller-manager pod log and pod previous log contains the error log "Unable to retrieve Node status":

E0223 18:31:51.557569 1 machineset_controller.go:883] "Unable to retrieve Node status" err="failed to create cluster accessor: failed to get lock for cluster: cluster is locked already" controller="machineset" controllerGroup="cluster.x-k8s.io" controllerKind="MachineSet" MachineSet="e2e-mycluster-v1-24-106-sks-upgrade/e2e-50a8u5-sks-upgrade-3m3w-workers-bkzcs" namespace="e2e-mycluster-v1-24-106-sks-upgrade" name="e2e-50a8u5-sks-upgrade-3m3w-workers-bkzcs" reconcileID=b9a3b2d2-00e9-4d0f-97b4-f2448292404d MachineDeployment="e2e-mycluster-v1-24-106-sks-upgrade/e2e-50a8u5-sks-upgrade-3m3w-workers" Cluster="e2e-mycluster-v1-24-106-sks-upgrade/e2e-50a8u5-sks-upgrade-3m3w" Machine="e2e-mycluster-v1-24-106-sks-upgrade/e2e-50a8u5-sks-upgrade-3m3w-workers-bkzcs-75tm4" node=""

This error causes the MD.Status.ReadyReplicas changes from 3 to 0 and after about 90s it will be changed back to 3. The reason is updateStatus() in machineset_controller.go ignores the error returned by getMachineNode() and treats the Node as not ready. In the mean time, KCP.Status.ReadyReplicas changes from 3 to 2 and back to 3 (after about only 8 seconds), and the reason might be kcp Reconcile() issues a requeue immediately after hitting ErrClusterLocked error.

Our code on top of CAPI watches MD.Status.ReadyReplicas and leads to an issue when MD.Status.ReadyReplicas changes from 3 to 0.

What did you expect to happen?

• MD.Status.ReadyReplicas should not change from 3 to 0 when (at least) hitting ErrClusterLocked error and even other errors, because the Nodes are ready actually.
• KCP.Status.ReadyReplicas should not change either when hitting ErrClusterLocked error.

Cluster API version

1.5.2

Kubernetes version

1.24.17

Anything else you would like to add?

To avoid MD.Status.ReadyReplicas change in this case, we can return error rather than contiune at
https://github.com/kubernetes-sigs/cluster-api/blob/v1.5.2/internal/controllers/machineset/machineset_controller.go#L882-L884 when the error is ErrClusterLocked (or even return error on any error).

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[chrischdi]

Note: CAPI v1.7 has fixes for the ErrClusterLocked error:

• 🐛 ClusterCacheTracker: Use RequeueAfter instead of immediate requeue on ErrClusterLocked to not have exponentially increasing requeue time #9810

Did we already test if this is still an issue with v1.7.0-rc.1 or so? (v1.7.0 will be released today)

[jessehu]

Thanks @chrischdi. That PR solves the reconcile latence when hitting ErrClusterLocked errors. This issue describes the issue "
MD.Status.ReadyReplicas changes from 3 to 0" when hitting ErrClusterLocked error for the first time.

[chrischdi]

Yeah just noted when reading code:

The fix it may not directly change the above behavior, but it would lead to retrying every minute.

[k8s-ci-robot]

This issue is currently awaiting triage.

CAPI contributors will take a look as soon as possible, apply one of the triage/* labels and provide further guidance.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jessehu]

/remove-lifecycle stale
There is a PR for resolving this issue, although with some concern: #10436

[fabriziopandini]

PTAL #10897

The new RemoteConnectionProbe condition and related --remote-connection-grace-period --remote-conditions-grace-period flags are laying the foundation for handling connection issues with the workload cluster in a consistent way across all the controllers.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[sbueringer]

I wonder if this still happens with v1beta2 conditions / new counter fields

[sbueringer]

/remove-lifecycle rotten

[jessehu]

I wonder if this still happens with v1beta2 conditions / new counter fields

cc @Levi080513 please help take a look!

[chrischdi]

I tested this scenario on main and am not able to reproduce this directly with the code on main.

I was able to reproduce it though by adding some latency to the clustercache:

diff --git a/controllers/clustercache/cluster_accessor.go b/controllers/clustercache/cluster_accessor.go
index 9c7e5944f..2d19cab87 100644
--- a/controllers/clustercache/cluster_accessor.go
+++ b/controllers/clustercache/cluster_accessor.go
@@ -250,6 +250,7 @@ func (ca *clusterAccessor) Connect(ctx context.Context) (retErr error) {
        }

        log.Info("Connecting")
+       time.Sleep(time.Second * 5)

        // Creating clients, cache etc. is intentionally done without a lock to avoid blocking other reconcilers.
        connection, err := ca.createConnection(ctx)

The error "error getting client" gets returned (from here) at to place:

https://github.com/kubernetes-sigs/cluster-api/blob/main/internal/controllers/machineset/machineset_controller.go#L1195

That leads to the being counted as not ready, but just because clustercache did not yet finish creating the connection yet.

So this issue is still valid for v1beta1 replicas.

v1beta2

Taking a look at v1beta2 replica fields: the same happens there, but the reason is a bit different.

In this case the Machine is counted as not ready, because the Machine's Ready v1beta2 condition flips to Unknown:

    - lastTransitionTime: "2025-02-06T12:15:03Z"
      message: '* NodeHealthy: ConnectionDown'
      observedGeneration: 3
      reason: ReadyUnknown
      status: Unknown
      type: Ready

This is because of the way we handle the Cluster not connected error here:

https://github.com/kubernetes-sigs/cluster-api/blob/main/internal/controllers/machine/machine_controller_status.go#L284-L294

[fabriziopandini]

@chrischdi thanks for investigating this!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sbueringer]

/remove-lifecycle stale

I still didn't find time to look at this