Velero data-mover fails behind proxy

[limboyet]

Hello!

I'm trying to do backups of data volumes with velero using CSI Snapshot Data Movement, backing up it to S3 like storage solution. The access to the backup storage location is using an HTTP proxy.

The problem is that velero saves the objects correctly to the backup location, but data-mover is not using proxy to do de PV data-mover.

I have observed that HTTP_PROXY environment variables are defined on the pod used for doing the data movement, but velero is trying to connect directly, without using proxy information.

This is the information of the created pod for doing the data movement, I have cleared IP's for privacy:

$ kubectl describe pod -n velero cluster-infra-midnight-20250402164114-2hs7w
Name:             cluster-infra-midnight-20250402164114-2hs7w
Namespace:        velero
Priority:         0
Service Account:  velero-server
Node:             k8s-infra-worker-02/10.249.54.85
Start Time:       Wed, 02 Apr 2025 18:41:28 +0200
Labels:           velero.io/data-upload=cluster-infra-midnight-20250402164114-2hs7w
                  velero.io/exposer-pod-group=snapshot-exposer
Annotations:      cni.projectcalico.org/containerID: 189797033623dd0ac609b4750375fcbd3ff4540011a5ebc73a2e6e4eacad410b
                  cni.projectcalico.org/podIP: 
                  cni.projectcalico.org/podIPs: 
Status:           Failed
IP:               10.25.22.212
IPs:
  IP:           10.25.22.212
Controlled By:  DataUpload/cluster-infra-midnight-20250402164114-2hs7w
Containers:
  410679b2-683e-47e2-b0ba-171bacae71b4:
    Container ID:  containerd://24559d999d854ff9c10d68e40a71d44de836d909b737f493c114d4bd4f2ed792
    Image:         velero/velero:v1.15.2
    Image ID:      docker.io/velero/velero@sha256:668467fdf39f3a610ed6f27431f38a6fbb6143a2ab302ad3e839b0074aaeba39
    Port:          <none>
    Host Port:     <none>
    Command:
      /velero
      data-mover
      backup
    Args:
      --volume-path=/410679b2-683e-47e2-b0ba-171bacae71b4
      --volume-mode=Filesystem
      --data-upload=cluster-infra-midnight-20250402164114-2hs7w
      --resource-timeout=10m0s
    State:          Terminated
      Reason:       Error
      Message:      Failed to run data path service for DataUpload cluster-infra-midnight-20250402164114-2hs7w: error to initialize data path: error to ensure backup repository sgrid-wordpress-test-kopia: failed to wait BackupRepository, errored early: backup repository is not ready: error to connect to backup repo: error to connect to storage: error retrieving storage config from bucket "velero-k8s-infra-backup": Get "https://*********/velero-k8s-infra-backup/k8s-infra-backup/kopia/wordpress-test/.storageconfig": dial tcp ********:443: connect: connection timed out
      Exit Code:    1
      Started:      Wed, 02 Apr 2025 18:41:39 +0200
      Finished:     Wed, 02 Apr 2025 18:41:39 +0200
    Ready:          False
    Restart Count:  0
    Environment:
      VELERO_NAMESPACE:                velero (v1:metadata.namespace)
      NODE_NAME:                        (v1:spec.nodeName)
      VELERO_SCRATCH_DIR:              /scratch
      AWS_SHARED_CREDENTIALS_FILE:     /credentials/cloud
      GOOGLE_APPLICATION_CREDENTIALS:  /credentials/cloud
      AZURE_CREDENTIALS_FILE:          /credentials/cloud
      ALIBABA_CLOUD_CREDENTIALS_FILE:  /credentials/cloud
      HTTPS_PROXY:                     http://************:8080
      HTTP_PROXY:                      http://************:8080
      NO_PROXY:                        10.0.0.0/8,192.168.0.0/16,172.16.0.0/12
    Mounts:
      /410679b2-683e-47e2-b0ba-171bacae71b4 from 410679b2-683e-47e2-b0ba-171bacae71b4 (rw)
      /credentials from cloud-credentials (rw)
      /host_plugins from host-plugins (rw)
      /host_pods from host-pods (rw)
      /scratch from scratch (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f4pdd (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  410679b2-683e-47e2-b0ba-171bacae71b4:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  cluster-infra-midnight-20250402164114-2hs7w
    ReadOnly:   false
  cloud-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sgrid
    Optional:    false
  host-pods:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  
  host-plugins:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins
    HostPathType:  
  scratch:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-f4pdd:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    BestEffort
Node-Selectors:               <none>
Tolerations:                  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:ScheduleAnyway when max skew 1 is exceeded for selector velero.io/exposer-pod-group=snapshot-exposer
Events:
  Type     Reason                  Age   From                     Message
  ----     ------                  ----  ----                     -------
  Warning  FailedScheduling        44s   default-scheduler        0/6 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
  Normal   Scheduled               42s   default-scheduler        Successfully assigned velero/cluster-infra-midnight-20250402164114-2hs7w to k8s-infra-worker-02
  Normal   SuccessfulAttachVolume  42s   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-705d4e29-177d-4b67-a471-85fb3bc591dc"
  Normal   Pulled                  32s   kubelet                  Container image "velero/velero:v1.15.2" already present on machine
  Normal   Created                 32s   kubelet                  Created container 410679b2-683e-47e2-b0ba-171bacae71b4
  Normal   Started                 32s   kubelet                  Started container 410679b2-683e-47e2-b0ba-171bacae71b4

Is there any other environment variable to be defined in order to use proxy to upload data to S3 storage?

Thanks a lot for your help.

[blackpiglet]

Those parameters should be used for the proxy setting.

      HTTPS_PROXY:                     http://************:8080
      HTTP_PROXY:                      http://************:8080
      NO_PROXY:                        10.0.0.0/8,192.168.0.0/16,172.16.0.0/12

As long as the setting is correct, the process, including the data mover, should use the proxy to communicate.
What the reported error?

[limboyet]

The reported error is:
Message: Failed to run data path service for DataUpload cluster-infra-midnight-20250402164114-2hs7w: error to initialize data path: error to ensure backup repository sgrid-wordpress-test-kopia: failed to wait BackupRepository, errored early: backup repository is not ready: error to connect to backup repo: error to connect to storage: error retrieving storage config from bucket "velero-k8s-infra-backup": Get "https://*********/velero-k8s-infra-backup/k8s-infra-backup/kopia/wordpress-test/.storageconfig": dial tcp ********:443: connect: connection timed out

The process is connecting direct to the storage server, instead of using proxy

[limboyet]

Hi all!

Is there any thought with that? Thanks in advance.

[rkthtrifork]

I have the same issue. How about we copy HTTPS_PROXY, HTTP_PROXY, and NO_PROXY environment variables from velero-server to the data mover jobs? Similar to the maintenance job https://github.com/vmware-tanzu/velero/blob/v1.16.2-rc.1/pkg/repository/maintenance/maintenance.go#L428

[limboyet]

The issue was resolved for me after upgrading to velero version v1.16.1.

What version are you using?

[rkthtrifork]

Im on v1.16.1. Do you know if the environment variables are present on your data mover jobs or if they just somehow know to go through the proxy? I think i had it working with kyverno injecting the environment variables, but i wouldnt think it would work without the environment variables. I also dont see any release notes about HTTP proxy since at least v1.12

[limboyet]

Yes, they are present. This is the output from describe of one of them:

$ kubectl describe pod -n velero ***-midnight-20250730112257-pb9rn
Name:             ***-midnight-20250730112257-pb9rn
Namespace:        velero
Priority:         0
Service Account:  velero-server
Node:             ***/10.10.100.86
Start Time:       Wed, 30 Jul 2025 13:23:14 +0200
Labels:           velero.io/data-upload=***-midnight-20250730112257-pb9rn
                  velero.io/exposer-pod-group=snapshot-exposer
Annotations:      cni.projectcalico.org/containerID: 09e643d6a92d2f0f68ece2933572da585814b72b61de826001ac0f2446dd56fb
                  cni.projectcalico.org/podIP: 10.25.24.23/32
                  cni.projectcalico.org/podIPs: 10.25.24.23/32
Status:           Running
IP:               10.25.24.23
IPs:
  IP:           10.25.24.23
Controlled By:  DataUpload/***-midnight-20250730112257-pb9rn
Containers:
  fed35e2a-c6bb-4061-9e8a-67e15701e6de:
    Container ID:  containerd://78686eded011690bb9cba9e69d7230f6f6cc1f660f4276163ddd666998daaa9d
    Image:         velero/velero:v1.16.1
    Image ID:      docker.io/velero/velero@sha256:c790429fcd543f0a5eed3a490e85a2c39bf9aefb8ce7ddbc7a158557745ab33f
    Port:          <none>
    Host Port:     <none>
    Command:
      /velero
      data-mover
      backup
    Args:
      --volume-path=/fed35e2a-c6bb-4061-9e8a-67e15701e6de
      --volume-mode=Filesystem
      --data-upload=***-midnight-20250730112257-pb9rn
      --resource-timeout=10m0s
    State:          Running
      Started:      Wed, 30 Jul 2025 13:23:20 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      VELERO_NAMESPACE:                velero (v1:metadata.namespace)
      NODE_NAME:                        (v1:spec.nodeName)
      VELERO_SCRATCH_DIR:              /scratch
      AWS_SHARED_CREDENTIALS_FILE:     /credentials/cloud
      GOOGLE_APPLICATION_CREDENTIALS:  /credentials/cloud
      AZURE_CREDENTIALS_FILE:          /credentials/cloud
      ALIBABA_CLOUD_CREDENTIALS_FILE:  /credentials/cloud
      HTTP_PROXY:                      http://***:8080
      HTTPS_PROXY:                     http://***:8080
      NO_PROXY:                        ***
    Mounts:
      /credentials from cloud-credentials (rw)
      /fed35e2a-c6bb-4061-9e8a-67e15701e6de from fed35e2a-c6bb-4061-9e8a-67e15701e6de (rw)
      /host_plugins from host-plugins (rw)
      /host_pods from host-pods (rw)
      /scratch from scratch (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-95fs8 (ro)

I have cleaned private information with "***", but you can see that environment variables are present.

[rkthtrifork]

Thank you. I does look like its supposed to inherit the environment variables of the node agent https://github.com/vmware-tanzu/velero/blob/main/pkg/exposer/csi_snapshot.go#L689

My node agent has http proxy environments variables so im unsure why theyre not being inherited 🤔 i assume its something on my end

[rkthtrifork]

I had a kyverno policy adding the proxy environment variables on the daemonset pods but not the daemonset itself and the data mover pods inherit the environment variables from the deamonset itself so they werent there