ClusterSource is immutable error

[gabschne]

I have a Redpanda cluster, which is set up using the operator in a Kubernetes env.

SASL auth is activated with SCRAM-SHA-512 as the mechanism.

I do use flux, therefore all topics, users and schemas are present as yaml files in a git repo, consumed and applied by the Flux kustomize operator.

I configured authentication for the mentions resources using a patch:

- target:
    kind: Topic|User|Schema
  patch: |-
    - op: add
      path: /spec/cluster
      value:
        staticConfiguration:
          kafka:
            brokers:
             - 'redpanda-0.redpanda.redpanda.svc.cluster.local:9093'
             - 'redpanda-1.redpanda.redpanda.svc.cluster.local:9093'
             - 'redpanda-2.redpanda.redpanda.svc.cluster.local:9093'
            tls:
              caCertSecretRef:
                name: 'redpanda-default-cert'
                key: 'ca.crt'
            sasl:
              username: admin
              passwordSecretRef:
                name: redpanda-admin-user
                key: password
              mechanism: 'SCRAM-SHA-512'
          admin:
            urls:
             - 'redpanda-0.redpanda.redpanda.svc.cluster.local:9644'
             - 'redpanda-1.redpanda.redpanda.svc.cluster.local:9644'
             - 'redpanda-2.redpanda.redpanda.svc.cluster.local:9644'
            tls:
              caCertSecretRef:
                name: 'redpanda-default-cert'
                key: 'ca.crt'
            sasl:
              username: admin
              passwordSecretRef:
                name: redpanda-admin-user
                key: password
              mechanism: 'SCRAM-SHA-512'
          schemaRegistry:
            urls:
             - 'redpanda-0.redpanda.redpanda.svc.cluster.local:8081'
             - 'redpanda-1.redpanda.redpanda.svc.cluster.local:8081'
             - 'redpanda-2.redpanda.redpanda.svc.cluster.local:8081'
            tls:
              caCertSecretRef:
                name: 'redpanda-default-cert'
                key: 'ca.crt'
            sasl:
              username: admin
              passwordSecretRef:
                name: redpanda-admin-user
                key: password
              mechanism: 'SCRAM-SHA-512'

However, as soon as I want to change something (for example I recently tried to add the schemRegistry section), the reconciliation (of Flux) fails because "ClusterSource is immutable":

{"level":"error","ts":"2025-03-10T17:02:02.092Z","msg":"Reconciliation failed after 441.574826ms, next try in 10m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{" name":"<REDACTED>","namespace":"flux-system"},"namespace":"flux-system","name":"<REDACTED>","reconcileID":"84144184-30b4-47e8-8629-08ffd440c785","revision":"main@sha1:60f9f329c07299ac3faede0614c24eb81a1a9bcc","error":"Topic/redpanda/<REDACTED> dry-run failed (Invalid): Topic.cluster.redpanda.com \"<REDACTED>\" is invalid: spec.cluster: Invalid value: \"object\": ClusterSource is immutable\n"}

Version info:

• appVersion: v2.3.7-24.3.6
• chartVersion: 0.4.40

[gabschne]

I tried different workarounds, e.g. extracting the schema registry patch for the Schema resources to it's own patch item, but it didn't help. Seems like my cluster is pretty much stuck now.

I know that completely deleting all topic resources does resolve the issue, however, this also deletes the topics in the cluster and I want to avoid that because they already contain a lot of migrated data.

Does someone have another idea how to workaround the issue?

[chrisseto]

Do you have a Redpanda CR created or are you managing the deployment of redpanda outside of the operator? (If so, I'd love to know why).

This is "expected" behavior but it was primarily targeted at the clusterRef parameter as changing the cluster the CR is pointing raises a lot of questions about the expected behavior and we opted to rule that out as a possible case. Perhaps we can be a bit less strict with the static configuration.

I tried different workarounds, e.g. extracting the schema registry patch for the Schema resources to it's own patch item, but it didn't help. Seems like my cluster is pretty much stuck now.

This should have worked, provided the Schema resources are getting created with the correct ClusterSource. Is flux applying the patches after the resources have been created?

[gabschne]

I have a Redpanda CR and managing the cluster with it. But I can't use the clusterRef field because SASL auth is enabled in the cluster and this would lead to authentication errors in the operator, if that's what your question was about.

I agree with having the clusterRef field being immutable is good thing, as configuring that is more or less a one-time operation and it protects from unwanted changes.

However, I also think that staticConfiguration should be mutable, as it's definitely more likely that valid and "non-questionable" changes will be made to it, such as changing the TLS settings, any of the secret names or, as in my case, wanting to configure the different parts one at a time.

This should have worked, provided the Schema resources are getting created with the correct ClusterSource. Is flux applying the patches after the resources have been created?

I checked my configs a second time and found the issue.