VPA: Pod memory limit exceeds recommendation and namespace quotas

[tspearconquest]

Which component are you using?:

/area vertical-pod-autoscaler

What version of the component are you using?:

Component version: 1.2.1

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
Client Version: v1.32.5
Kustomize Version: v5.5.0
Server Version: v1.33.1

What environment is this in?: Azure

What did you expect to happen?: VPA webhook would apply limits less than the maxAllowed value

What happened instead?: VPA webhook applied a limit far exceeding the maxAllowed value of the VPA resource and also exceeding the namespace limits.memory LimitRange value for pods and the namespace limits.memory ResourceQuota value

How to reproduce it (as minimally and precisely as possible): Just applied the VPA, waited for a few minutes, and then restarted the targetRef deployment

Anything else we need to know?:

Does VPA set limits at this time? I see earlier in #2747 that some years ago, VPA did not back then support setting limits, but I'm seeing odd behavior for a resource I have managed by VPA.

My VPA as you can see below has maxAllowed.memory configured to 1536Mi (1.5Gi):

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: source-controller-platform-admin
  namespace: flux-system
spec:
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      controlledResources:
      - cpu
      - memory
      maxAllowed:
        cpu: 1000m
        memory: 1536Mi
      minAllowed:
        cpu: 50m
        memory: 768Mi
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: source-controller-platform-admin
  updatePolicy:
    minReplicas: 1
    updateMode: Auto

Namespace limit range allows up to 2560Mi per pod and per container (1 container per pod) and specifies defaults less than that to limit any pods not managed by VPA:

apiVersion: v1
kind: LimitRange
metadata:
  name: flux-system
  namespace: flux-system
spec:
  limits:
  - default:
      cpu: "1"
      ephemeral-storage: 1280Mi
      memory: 1280Mi
    defaultRequest:
      cpu: 500m
      ephemeral-storage: 1280Mi
      memory: 1280Mi
    max:
      cpu: "1"
      ephemeral-storage: 2560Mi
      memory: 2560Mi
    type: Container
  - max:
      cpu: "1"
      ephemeral-storage: 2560Mi
      memory: 2560Mi
    type: Pod

Namespace resource quota allows up to 25Gi total memory usage in the namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: limit-total-namespace-cpu-memory-storage-usage
  namespace: flux-system
spec:
  hard:
    limits.ephemeral-storage: 25Gi
    limits.memory: 25Gi
    requests.cpu: "20"
    requests.ephemeral-storage: 25Gi
    requests.memory: 25Gi

My deployment and replicaset have memory defined as:

resources:
  limits:
    memory: 1536Mi
  requests:
    cpu: 50m
    memory: 64Mi

Yet, when the pod is created by the replicaset, I get this error in the replicaset events (it's a singleton service but there are other singleton pods in the same namespace using some of the memory):

message: 'pods "source-controller-platform-admin-74dfbb9c79-7k7w8" is forbidden:
  exceeded quota: limit-total-namespace-cpu-memory-storage-usage, requested: limits.memory=18Gi,
  used: limits.memory=25440Mi, limited: limits.memory=25Gi'

Namespace memory usage currently without this pod:

limits.memory: 23904Mi/25Gi

This is from another pod in the same namespace targeted by a different VPA also got 18Gi limit applied by VPA recommendation and took up a large percentage of the available limits.memory quota.

My VPA doesn't seem to be recommending limits of 18Gi for this pod as seen below:

status:
  conditions:
  - lastTransitionTime: "2025-08-03T15:06:50Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: manager
      lowerBound:
        cpu: 50m
        memory: 768Mi
      target:
        cpu: 63m
        memory: 768Mi
      uncappedTarget:
        cpu: 63m
        memory: "764046746"
      upperBound:
        cpu: 628m
        memory: 1536Mi

And nothing I have configured in the namespace should be configuring the limit that high (18Gi), as seen with the limit range resource defined above.

However, if I remove the VPA, the pod gets created with the value from the deployment and all is well; so it clearly seems to be the VPA mutating the pod at admission causing it to get a limit value set to 18Gi which exceeds the namespace resource quota.

So again I just want to confirm if VPA now supports setting limits and has any issues with this?

This is the AKS managed supported installation of VPA. The recommender pod runs this image:

mcr.microsoft.com/oss/v2/kubernetes/autoscaler/vpa-recommender:v1.2.1-1

[tspearconquest]

I think I've just found the issue.

AKS docs link: https://learn.microsoft.com/en-us/azure/aks/use-vertical-pod-autoscaler#extra-recommender-for-vertical-pod-autoscaler

While I'm not using an extra recommender, this was found in the section for an extra recommender.

controlledValues allows you to choose whether to update the container's resource requests using theRequestsOnly option, or by both resource requests and limits using the RequestsAndLimits option. The default value is RequestsAndLimits. If you use the RequestsAndLimits option, requests are computed based on actual usage, and limits are calculated based on the current pod's request and limit ratio.

For example, if you start with a pod that requests 2 CPUs and limits to 4 CPUs, VPA always sets the limit to be twice as much as requests. The same principle applies to Memory. When you use the RequestsAndLimits mode, it can serve as a blueprint for your initial application resource requests and limits.

Given that my pods starts with:

resources:
  limits:
    memory: 1536Mi
  requests:
    cpu: 50m
    memory: 64Mi

The ratio is 1:24, so when the VPA then tries to apply the recommended requests of 768Mi, it tries to maintain the 1:24 ratio and is setting the limits to 18432Mi (18Gi)

This is a bit counterintuitive behavior(IMHO) and it's preferable for me to be able to explicitly define the memory requests:limit ratio in the VPA resource. It would be great to tell the VPA to just ignore the starting limits and requests and have the requests:limit ratio be 1:1 always, because for this service (FluxCD's "source-controller") it comes with the requests and limits pre-defined, and removing them from the deployment causes the namespace limit range to apply to the pods at first due to the webhook run order which I can't control because this is AKS. I can change them in the deployment with kustomize so that the desired ratio is met, but it definitely needs to be documented clearly that VPA will maintain the ratio defined between requests and limits in the targetRef.

[adrianmoisey]

This is a bit counterintuitive behavior(IMHO) and it would be more preferable to explicitly define the memory requests:limit ratio in the VPA resource

This is not a bad idea. Having the ratio settable as part of the API may be helpful to let people know that the ratio is how the VPA works.

but it definitely needs to be documented clearly that VPA will maintain the ratio defined between requests and limits in the targetRef

We have it documented here. If you have any suggestions as to how we can improve it, please make a PR:
https://github.com/kubernetes/autoscaler/blob/c7a51426e28385824ca3c05be929d80854e6c39b/vertical-pod-autoscaler/docs/features.md#limits-control

[tspearconquest]

This is not a bad idea. Having the ratio settable as part of the API may be helpful to let people know that the ratio is how the VPA works.

For the specific purpose of providing information, I'd say having a field in the status of the VPA indicating the exact ratio it's following to set the limit would suffice to make it clear, but yes having it configurable is preferable over implicitly using the ratio from the targetRef; though the ratio from targetRef is still a useful default when a value is not provided explicitly.

We have it documented here.

Oh, ok I see it here.

This is very useful info toward the error I was getting:

If limit range conflicts with VPA resource policy, VPA will follow VPA policy (and set values outside the limit range).

I happened to skip over the features doc looking for similar info in the known limitations doc instead. Helps to read everything I suppose! :)

[adrianmoisey]

If limit range conflicts with VPA resource policy, VPA will follow VPA policy (and set values outside the limit range).

I happened to skip over the features doc looking for similar info in the known limitations doc instead. Helps to read everything I suppose! :)

This particular one is something that people often trip over, so it's possibly worth us trying to highlight this behaviour somehow.

[tspearconquest]

Indeed. Would be happy to add it to known limitations, but am also open to other ideas. What do you think about adding the ratio to status?

Example:

status:
  recommendation:
    containerRecommendations:
    - containerName: manager
      controlledValues: RequestsAndLimits # New key making it clear what values are controlled
      lowerBound:
        cpu: 62m
        memory: 768Mi
# New field showing ratios for each controlled resource and which only shows if controlled values is RequestsAndLimits
      ratios:
        cpuRequestsToLimits: 1:5
        memoryRequestsToLimits: 1:10
# End new field
      target:
        cpu: 78m
        memory: 768Mi
      uncappedTarget:
        cpu: 78m
        memory: "764046746"
      upperBound:
        cpu: 961m
        memory: 1536Mi

[adrianmoisey]

What do you think about adding the ratio to status?

I'm not too sure about this one. I really like it, I think it's a good idea to bubble the information up to the user.

What worries me is that the admission-controller portion is responsible for calculating the resources for a Pod on admission. The requests/limits could be set by the Pod's controller, or another admission controller plugin (ie: LimitRanger). Having the admission-controller update the status field might not be ideal, although not impossible. But it's really just saying what the last "observed" ratio.
This makes me wonder if it's worth adding to an Event or to the annotation on the Pod that the VPA creates.

Both of these options are potentially a little more "hidden" from the user than the status idea though.

[tspearconquest]

The pod annotation would be a good place for it as well. It'd be right alongside the annotation LimitRanger adds when it mutates the pod, so it could help track down where the ratio originates when the user sees both annotations.

[tspearconquest]

I agree each pod should stay in its own lane, as it were; not having the admission controller updating the VPA status field since the recommender does that already.

You mentioned that the admission controller calculates the resources assigned; is it the case that the admission controller updates the values based on the recommender's recommendations? If the recommender could recommend the ratio rather than the admission controller picking one, that would solve the issue.

Would it make sense to have the ratio calculation come from recommender? Perhaps, then recommender could take the limit range into account and recommend a ratio for the admission controller that's dynamically adjusted to fit within the limit range values, at which point the recommender can simply update the VPA status field itself with the ratio it calculated, and the admission controller can update the pod annotation as well. Surface the info in both locations to maximize visibility. :)

Just my 2c.

[adrianmoisey]

You mentioned that the admission controller calculates the resources assigned; is it the case that the admission controller updates the values based on the recommender's recommendations?

Correct. The recommendation for the requests comes from the recommender, which the admission-controller reads to make its decision on pod admission.

Would it make sense to have the ratio calculation come from recommender? Perhaps, then recommender could take the limit range into account and recommend a ratio for the admission controller that's dynamically adjusted to fit within the limit range values, at which point the recommender can simply update the VPA status field itself with the ratio it calculated, and the admission controller can update the pod annotation as well. Surface the info in both locations to maximize visibility. :)

Agreed, however, changing behaviour now is a backwards incompatible change. I don't think there's a way for the recommender to know what admission plugins a user has, or how those could change the requests/limits before the Pod gets to the admission-controller.

My thinking is a "ratio" field on the VPA object somewhere, if set, the admission-controller uses that ratio. If unset, it falls back to the current behaviour.

[raywainman]

+1 that this trips people up - if this is desired enough, we could add a ratio field in the API though I would like to understand a bit more about how folks would use it.

For example, would it just be better to capture a memoryLimit in maxAllowed?

And if exploring the ratio option, how do we capture the various scenarios? Is 1:1 the default? I almost wonder if folks would just prefer an option to preserve the limit-request buffer (I'm not exactly sure how we would capture this but something to think about).

[iamzili]

@tspearconquest - let me paste this section of text as well about setting a global maximum in VPA, I think it could be interesting for you: link

My understanding is that currently, if we want to change the request-to-limit ratio for any reason, we need to manually update the resource requests and limits in the pod spec. I think it would make more sense to do this at the VPA level, because if we enable VPA for a workload, I assume we don't want to update the resource requests and limits manually just to change the ratio. So I think what @adrianmoisey suggests seems like a reasonable solution to me.

@raywainman - from your message, are you suggesting that the VPA could automatically populate the maxAllowed field - for example, based on the cluster's max allocatable resources or the namespace's LimitRange policy? btw I think the default should be what Adrian suggested, which is to maintain the initially defined request-to-limit ratio - this is the current behaviour as well

[tspearconquest]

@tspearconquest - let me paste this section of text as well about setting a global maximum in VPA, I think it could be interesting for you: link

Thanks! I might have to run a secondary copy of the recommender pod to take advantage of this because I'm using the Azure managed VPA; at least that's something of an option their docs suggest is viable. I appreciate this.

My understanding is that currently, if we want to change the request-to-limit ratio for any reason, we need to manually update the resource requests and limits in the pod spec. I think it would make more sense to do this at the VPA level, because if we enable VPA for a workload, I assume we don't want to update the resource requests and limits manually just to change the ratio. So I think what @adrianmoisey suggests seems like a reasonable solution to me.

I agree. That's pretty accurately summing up the goal for me as well. The pod in question is part of Flux, and they provide default values that amounts to a ratio of 1:16 (64Mi requests, 1Gi limits) - so being able to customize the ratio via VPA would simplify the deployment of the Flux services because I don't have to additionally kustomize patch the requests and limits of the deployment to set the initial ratio VPA follows.

[AndreasDavour]

This tripped me up as well. If we can't configure how vpa sets the limit, a note in the status would help a lot while debugging, but a settable ratio would be even better.