Trying to add a schema with schema CRD fails in unexpected error

[gabschne]

I have a Redpanda cluster, which is set up using the operator in a Kubernetes env.

SASL auth is activated with SCRAM-SHA-512 as the mechanism.

I do use flux, therefore all resources are present as yaml files in a git repo, consumed and applied by the Flux kustomize operator.

I tried to add schemas, and while the resources are readable in the K8s cluster, the synced state is false and they have the following error:

Status:
   Conditions:
     Last Transition Time:  2025-03-19T02:35:42Z
     Message:               Error: unable to PUT "http://redpanda-2.redpanda.redpanda.svc.cluster.local:8081/config/<REDACTED>": Put "http://redpanda-2.redpanda.redpanda.svc.cluster.local:8081/config/<REDACTED>": EOF
     Observed Generation:   1
     Reason:                UnexpectedError
     Status:                False
     Type:                  Synced
   Observed Generation:     1
   Schema Hash:         

The CR for the schema(s) look like:

Name:         <REDACTED>
Namespace:    redpanda
Labels:       kustomize.toolkit.fluxcd.io/name=l2
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>
API Version:  cluster.redpanda.com/v1alpha2
Kind:         Schema
Metadata:
  Creation Timestamp:  2025-03-19T12:52:19Z
  Finalizers:
    operator.redpanda.com/finalizer
  Generation:        1
  Resource Version:  29396792
  UID:               1a89a5fd-7eb1-4094-81e9-12e228e356fe
Spec:
  Cluster:
    Static Configuration:
      Schema Registry:
        Sasl:
          Mechanism:  SCRAM-SHA-512
          Password Secret Ref:
            Key:     password
            Name:    redpanda-admin-user
          Username:  admin
        Tls:
          Ca Cert Secret Ref:
            Key:   ca.crt
            Name:  redpanda-default-cert
        Urls:
          redpanda-0.redpanda.redpanda.svc.cluster.local:8081
          redpanda-1.redpanda.redpanda.svc.cluster.local:8081
          redpanda-2.redpanda.redpanda.svc.cluster.local:8081
  Compatibility Level:  BackwardTransitive
  Schema Type:          avro
  Text:                 {
    "type": "record",
    "name": "<REDACTED>",
    "namespace": "<REDACTED>",
    "doc": "<REDACTED>",
    "fields": [
      {
        "name": "eventId",
        "type": "string"
      },
      {
        "name": "packageId",
        "type": "string",
        "doc": "The packageId of the content that has been played."
      },
      [...]
    ]
  }

The operator logs the following errors:

{"level":"info","ts":"2025-03-19T14:00:51.390Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"schema","controllerGroup":"cluster.redpanda.com","controllerKind":"Schema","Schema":{"name":"<REDACTED>","namespace":"redpanda"},"namespace":"redpanda","name":"<REDACTED>","reconcileID":"88b0a824-04cc-4d3b-a3a8-da79610727f0"}
{"level":"error","ts":"2025-03-19T14:00:51.390Z","msg":"Reconciler error","controller":"schema","controllerGroup":"cluster.redpanda.com","controllerKind":"Schema","Schema":{"name":"<REDACTED>","namespace":"redpanda"},"namespace":"redpanda","name":"<REDACTED>","reconcileID":"88b0a824-04cc-4d3b-a3a8-da79610727f0","error":"unable to PUT \"http://redpanda-2.redpanda.redpanda.svc.cluster.local:8081/config/<REDACTED>\": Put \"http://redpanda-2.redpanda.redpanda.svc.cluster.local:8081/config/<REDACTED>\": EOF"}

I followed the guide at https://docs.redpanda.com/current/manage/kubernetes/k-schema-controller/#create-a-schema.

And I got the static configuration part from https://docs.redpanda.com/current/reference/k-crd/#k8s-api-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-staticconfigurationsource. I found no documentation describing how to configure auth, but I guess kafka and admin are not needed in schemas, but only the schemaRegistry section.

It worked fine for topic and user resources.

I don't think it has anything to do with the schema itself, but rather with the enabled SASL authentication. Adding the schemas without SASL enabled (in a local test cluster) works fine.

Trying to remove/delete the schema CRs didn't work either, because operator.redpanda.com/finalizer is set as a finalizer, I had to remove it forcefully from the list of finalizers in every schema resource in the cluster.

Version info:

• operator chartVersion: 0.4.40
• redpanda chartVersion: 0.5.20

[andrewstucki]

So, one thing that immediately jumps out is that the PUT is to: http://redpanda-2.redpanda.redpanda.svc.cluster.local:8081/config/... rather than https when you've specified a TLS certificate. With both the schema registry static configuration block as well as the redpanda admin API static configuration block, both of those are meant to be HTTP-based URLs with the scheme included rather than just addresses with ports. My first guess is that the EOF is coming from trying to talk to an HTTPs listener over HTTP.

Also, is there a reason not to use clusterRef?