Aws::CloudWatch::Client doesn't automatically refresh credentials

[Roy-Gal-Git]

Describe the bug

We use the sidekiq-cloudwatchmetrics gem to send Sidekiq-related metrics to CloudWatch. Occasionally, we encounter the following error:

Aws::CloudWatch::Errors::ExpiredToken: The security token included in the request is expired

Upon investigation, we found that the gem initializes an Aws::CloudWatch::Client, which uses Aws::InstanceProfileCredentials. These credentials are supposed to auto-refresh before expiration. However, the error indicates that the token is not being refreshed in time, causing the metrics to fail. Once the error starts, it repeats, and the metrics fail to recover.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The Aws::InstanceProfileCredentials should automatically refresh the security token before expiration (approximately 5 minutes before expiry) and ensure that the put_metric_data API call succeeds without errors. Metrics should continue to be sent to CloudWatch without interruption.

Current Behavior

The Aws::CloudWatch::Client raises Aws::CloudWatch::Errors::ExpiredToken errors when attempting to send metrics.

Once this error occurs:

1. The metrics stop being sent to CloudWatch.
2. The error persists until the process is restarted or manually handled.
3. The auto-refresh mechanism in Aws::InstanceProfileCredentials does not appear to trigger correctly in this context.

Reproduction Steps

Configure the sidekiq-cloudwatchmetrics gem in a Sidekiq setup that uses an IAM role (e.g., on EC2 or ECS).
You can find the configuration I used here:
Allow the process to run for an extended period, relying on auto-refreshable credentials from Aws::InstanceProfileCredentials.
Observe that after some time, the following error may appear:
Aws::CloudWatch::Errors::ExpiredToken: The security token included in the request is expired

Possible Solution

No response

Additional Information/Context

Here are some useful links for more context regarding my problem:

• The issue I opened in the sidekiq-cloudwatchmetrics repo Reoccurring Aws::CloudWatch::Errors::ExpiredToken errors sj26/sidekiq-cloudwatchmetrics#48
• The PR I opened to fix the problem in the sidekiq-cloudwatchmetrics repo https://github.com/sj26/sidekiq-cloudwatchmetrics/pull/49/files

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-cloudwatch

Environment details (Version of Ruby, OS environment)

Ruby - 3.2.5, OS - Debian GNU/Linux 12 (bookworm), Docker image - ruby:3.2.5-slim, sidekiq-cloudwatchmetrics - 2.6.0

[alextwoods]

You are correct that the InstanceProfileCredentials should be auto refreshing (if they are within 5 minutes of expiration).

We added "static stability" logic awhile back that allow expired credentials to be used after their expiration when the underlying credentials service isn't refreshing credentials on the hosts (eg: an outage). Are you seeing any logs like Attempting credential expiration extension due to a credential service availability issue?

Have you tested your PR that does the explicit credentials refresh - does that fix the issue for you?

[Roy-Gal-Git]

Hi @alextwoods,

I’m unable to reproduce the error, so I can’t verify whether the issue has been resolved. Even if my PR makes it to production, I won’t be able to confirm if it fixes the issue for me, as the error occurs randomly across different production environments.

Are you seeing any logs like Attempting credential expiration extension due to a credential service availability issue?

No, I didn't find any logs such as what you suggested.
I tried looking for anything containing the words 'credential' and 'attempting' (besides the exact match of the full message you sent) but I didn't find anything either.

[mullermp]

This may be a naive question, but are you certain that the other gem is resolving instance profile credentials, and credentials are not set globally or in some other way? Reading your other thread, it looks like you tried creating a client on your own, and the credentials defaulted to instance profile credentials (which is last in the chain) but it's possible in your app in production, it is set to something else. I would actually output an instance of your credentials provider after deploying somehow.

[Roy-Gal-Git]

Hi @mullermp,

are you certain that the other gem is resolving instance profile credentials, and credentials are not set globally or in some other way?

The other gem's CloudWatch publisher requires a client argument, and users can determine how this client is configured. In my case, I explicitly pass Aws::CloudWatch::Client.new(region: REGION_PLACEHOLDER), which resolves to instance profile credentials in the production environment.

I would actually output an instance of your credentials provider after deploying somehow

I shared this in the other thread as well, but here’s the truncated output of @client.config (where @client is the CloudWatch client I pass to the gem’s publisher) from a production environment:

#<struct
	credentials=
		#<Aws::InstanceProfileCredentials:0x00007f78ee260660
			...
			@credentials=#<Aws::Credentials access_key_id="PLACEHOLDER_ACCESS_KEY_ID">,
			...>
	...
	secret_access_key=nil
	session_token=nil
	...>

This confirms that the client resolves to Aws::InstanceProfileCredentials, as expected in this setup.

[mullermp]

Yes I saw that. Try passing an instance of instance profile credentials explicitly to the client in your production code. Looking at the gem source, I don't think any patch is necessary. The SDK will already refresh and retry if necessary.

[Roy-Gal-Git]

@mullermp How come it doesn't already refresh and retry if it does use an instance of instance profile credentials?

[mullermp]

It does and should. You can verify that by fetching instance credentials from irb or pry from your host and inspect them after accessing them before expiration. How did you verify the credentials class in production? Using a rails console or print statement? Or through local testing? By passing in credentials explicitly, we can be sure that is what is used. What version of aws-sdk-cloudwatch and aws-sdk-core are you using?

[Roy-Gal-Git]

How did you verify the credentials class in production?

Rails console. I initialized an Aws::CloudWatch::Client object the way I do for thesidekiq-cloudwatchmetrics publisher and performed the checks on that object. You're saying the client credentials could differ when tested from rails console?

What version of aws-sdk-cloudwatch and aws-sdk-core are you using?

aws-sdk-cloudwatch (1.108.0)
aws-sdk-core (3.214.0)

[mullermp]

They shouldn't differ. Trust but verify, but did you pass rails env as production in your console? Are you setting global configuration anywhere such as Aws.config[key]?

Otherwise I'm not sure what may be going on. I'm inclined to believe it's something with your environment because if those credentials don't refresh then there would be a lot more noise and our internal usage would stop working.

[Roy-Gal-Git]

did you pass rails env as production in your console?

We use a setup where the Rails environment name matches the server name (e.g., us, eu). The correct Rails env is automatically passed to the Rails console as an exported environment variable.

Are you setting global configuration anywhere such as Aws.config[key]?

No, we are not setting Aws.config[key] anywhere in our application.

if those credentials don't refresh then there would be a lot more noise and our internal usage would stop working.

Interestingly, the only component affected is the sidekiq-cloudwatchmetrics publisher. The app continues to function as expected, but metrics stop being sent to CloudWatch, which triggers an alarm after a short while.

[alextwoods]

The app continues to function as expected, but metrics stop being sent to CloudWatch, which triggers an alarm after a short while.

Do you use other AWS services in your application? If so, its interesting that you would get expired token issues only in cloudwatch.

Do you know if you are using IMDS V1 or V2? And do you know if you've disabled V1 through configuration (example the AWS_EC2_METADATA_V1_DISABLED env variable)? There are some differences in the refresh process for credentials from V1 and V2.