bug: is there a memory leak? If so, fix?

[davidxia]

What happened:

Many users have reported in aws/containers-roadmap#2605 that they suspect a memory leak. I also observe monotonically increasing memory usage that eventually results in OOMKill of the K8s container after it's been running for about 20 days.

What you expected to happen:

Memory usage isn't monotonically increasing and no OOMs.

How to reproduce it (as minimally and precisely as possible):

Run the eks-node-monitoring-agent as instructed and just observe its memory and OOMKills.

Environment:

• AWS Region: us-east-1
• Instance Type(s): c6in.2xlarge
• Cluster Kubernetes version: v1.32.7-eks-b707fbb
• Node Kubernetes version: v1.32.7-eks-3abbec1
• EKS Node Monitoring Agent version: v1.4.0-eksbuild.2 (latest of this writing)

[shvbsle]

Thank you for flagging this. Have you noticed this behavior across different node types or just on c6in.2xlarge instances?

[davidxia]

I've also seen OOMKills on

c7a.4xlarge
c6a.16xlarge
c6a.2xlarge

[cilindrox]

@shvbsle I've noticed it on several types of instances, both x86 and ARM: aws/containers-roadmap#2605 (comment)

[darkhelmet]

I get an OOMKilled/CrashLoopBackoff loop as soon as I try to create a NodeDiagnostic resource to get logs. m7i.2xlarge

[shvbsle]

After some profiling, I suspect this is not a memory leak but rather a memory-spike from some code-paths that pushes the pod beyond memory limits. The default memory limits set might not be right for everyone. @davidxia @cilindrox @darkhelmet have you folks tried tuning the memory limits for nodeAgent to see if it allays the OOMKills?

Something like:

nodeAgent:
  resources:
    requests:
      cpu: 10m
      memory: 30Mi
    limits:
      cpu: 250m
      memory: 200Mi # double it or tune this value

[alexkuretz]

I've seen this a lot, particularly on larger C/R instance types (8x-24xlarge). I also see spiky behavior as mentioned above. We use Karpenter/Keda for scaling workloads, in case that's relevant.

I'm trying increasing the memory limit from 100 to 150 to see if it helps. I use terraform to configure the add-ons using the EKS module:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.2.0"
...
  addons = {
    eks-node-monitoring-agent = {
      most_recent                 = true
      resolve_conflicts_on_create = "OVERWRITE"
      configuration_values = jsonencode({
        monitoringAgent = {
          resources = {
            requests = {
              cpu    = "10m"
              memory = "30Mi"
            }
            limits = {
              cpu    = "250m"
              memory = "150Mi"
            }
          }
        }
      })
    }
  }
...
}

[davidxia]

@shvbsle Thanks for looking into this. I doubled the memory limit to 200Mi. Will keep you posted.

[davidxia]

@shvbsle This 4-day old Pod is showing this monotonically increasing memory usage behavior. It's increasing very slowly, but I still wonder if this is expected.

[nathanmcgarvey-modopayments]

After some profiling, I suspect this is not a memory leak but rather a memory-spike from some code-paths that pushes the pod beyond memory limits. The default memory limits set might not be right for everyone. @davidxia @cilindrox @darkhelmet have you folks tried tuning the memory limits for nodeAgent to see if it allays the OOMKills?

Something like:

nodeAgent:
resources:
requests:
cpu: 10m
memory: 30Mi
limits:
cpu: 250m
memory: 200Mi # double it or tune this value

FWIW: the memory usage is almost never at or below 30Mi in my environment(s). Usually 50Mi or less, but those requests and limits may be too low for most folks, not just "not right for everyone". (I'm using c7i.xlarge)

Also, I'm wondering if the cpu limits are putting backpressure on the go garbage collector so that it cannot release memory fast enough during spike events. Many sources recommend not putting a cpu limit on containers unless you have licensing concerns or really specific use-cases. Similarly, except in the case of true spikey workloads (external user input, etc.), memory requests/limits are frequently the same. For a monitoring agent, that seems odd, but, since source code isn't public, it's really hard to inspect and help.

[MartinEmrich]

+1. Such a node monitoring agent should have a footprint as low as possible. A spike of 200MiB, even for a few seconds, risks pushing out "legitimate" workloads on the same node.

When running smaller nodes with 8GB RAM for example, 200MiB is a cost-relevant fraction of that memory.

The monitoring agent 1.3.0 did not exhibit this behaviour (appeared here on 1.4.0), so the previous version was more conservative with memory.

[kunhwiko]

In our case, we typically use R instances (large ~ 32xlarge) and we seem to be observing memory increases over time as well.

The default 100Mi was too small for us, so we bumped up the agent's memory to 500Mi and observed across 8 days. This is the pattern we're seeing:

---

From our internal analysis, we don't think this is due to CPU backpressure and issues with garbage collection (though we could be wrong here):

• Our metrics are showing that the average CPU usage never really gets anywhere close to CPU limits.
• GC doesn't seem to be consuming a whole lot of CPU time, though we haven't captured during spike events.

Oct 16, 2025
# NumGC = 17223
# GCCPUFraction = 0.0003131606860229997
 
Oct 19, 2025
# NumGC = 22213
# GCCPUFraction = 0.00028453520945639205
 
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.3091e-05
go_gc_duration_seconds{quantile="0.25"} 0.000103222
go_gc_duration_seconds{quantile="0.5"} 0.000121902
go_gc_duration_seconds{quantile="0.75"} 0.000171182
go_gc_duration_seconds{quantile="1"} 0.101457755

---

The only place where we see inuse heap memory increasing over time seems to be *golang.a2z.com/EKSNodeMonitoringAgent/internal/monitor/storage.(StorageMonitor).checkIODelays. I'm not sure if this is intended behavior or if others are observing the same in their pprof analysis. Below is a flamegraph that shows this:

Flamegraph for Cluster A:

Flamegraph for Cluster B:

I am sharing the pprof sample (i.e. goroutine, heap, runtime memstat samples) that we captured over 8 days:
pprof-sample-oct-2025.zip

[shvbsle]

Thanks for the investigation! We have a release coming up that will allay this issue. Will keep you folks posted.

[pnowy]

Hello, any ETA of release the fix? We have the memory issue on multiple clusters and just wondering if we should wait or fix on our own with resource/requests as temporary workaround. Thanks