Upgrading server cluster to v1.10.3 unable to discover fellow peers

[bfqrst]

Nomad version

Nomad v1.10.3
BuildDate 2025-07-08T18:46:27Z
Revision fde0bf2

Operating system and Environment details

Ubuntu Noble
Linux auto-nomad-dwt 6.8.0-1030-aws #32-Ubuntu SMP Wed May 28 19:33:40 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

Issue

Upgrading cluster on AWS via rolling release (ASG) results in new cluster peers unable to find themselves.

Reproduction steps

Have a AWS ASG ready with 3 members. Build new image via Packer with v1.10.3 in it. Cycle ASG.

Expected Result

Running cluster being able to find fellow cluster members.

Actual Result

Jul 09 10:20:34 auto-nomad-dwt nomad[597]: 2025-07-09T10:20:34.579+0200 [ERROR] agent.joiner: discovering join addresses failed: agent_mode=server join_config="provider=aws region=eu-central-1 tag_key=nomad-servers tag_value=nomad-auto addr_type=private_v4" error="discover-aws: DescribeInstancesInput failed: operation error EC2: DescribeInstances, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post \"https://ec2.eu-central-1.api.aws/\": AuthorizedOnly"

Reverting new member back to v1.10.2 resolves the issue.

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[bfqrst]

It appears to be go-discover v1.1.0 related...

1.1.0 (2025-06-12)
Improvements

• AWS provider: enable dual-stack support by default, allowing the use of IPv6 endpoints wherever required. Adding support for regexp, version, and lexical ordering constraints #271

I was fairly positive that I haven't seen a .aws Domain in the logs... Since we've got a somewhat restrictive corporate proxy it might necessary to add .aws to the NO_PROXY list.

[bfqrst]

So TIL, the dual-stack endpoint appears to be a different endpoint than the classic non-dual stack endpoint, even if it resolves to IPv4! If you're in a regulated environment AND your account doesn't have an IGW and everything is routed to a TGW AND the dual-stack endpoint isn't available -> no dice!

I've tried to set AWS_USE_DUALSTACK_ENDPOINT= false in the Nomad systemd file to no avail. If somebody has an idea how to tell go-discover or AWS to use the classic endpoint, I'm all ears!

For those who are in a regulated environment where you don't control the IGW or TGW plus a proxy, better make sure the dual-stack situation works! Of course if one doesn't rely on automatic cloud clustering and joins the server members some other way, you might be golden too...