Skip to content

Fix intermittent test failure #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 4, 2025
Merged

Fix intermittent test failure #91

merged 3 commits into from
Jun 4, 2025

Conversation

jemiahw
Copy link
Collaborator

@jemiahw jemiahw commented Jun 4, 2025

A number of Etre tests verify the server side metrics after performing some operation (create, update, delete, etc.). The existing assertions assume latency will be 0. Usually it is, but occasionaly on the GitHub servers running automated tests on the PR, the latency will be a few milliseconds. This causes the test to fail since the latency metric will have a value that doesn't match the 0 in the assertion.

To fix this, I've added a method "fixLatencyMetric". This method finds any latency metrics in the "expect" list and updates the value to match the corresponding metric from the "actual" list. To make sure the latency metric value is not garbage, fixLatencyMetrics also takes a max latency and asserts that the actual latency is between 0 and max. This way we still check the latency metric has some reasonable value, but we won't fail the test if the latency is a few milliseconds instead of 0.

A number of Etre tests verify the server side metrics after performing some operation (create, update, delete, etc.). The existing assertions assume latency will be 0. Usually it is, but occasionaly on the GitHub servers running automated tests on the PR, the latency will be a few milliseconds. This causes the test to fail since the latency metric will have a value that doesn't match the 0 in the assertion.

To fix this, I've added a method "fixLatencyMetric". This method finds any latency metrics in the "expect" list and updates the value to match the corresponding metric from the "actual" list. To make sure the latency metric value is not garbage, fixLatencyMetrics also takes a max latency and asserts that the actual latency is between 0 and max. This way we still check the latency metric has some reasonable value, but we won't fail the test if the latency is a few milliseconds instead of 0.
@jemiahw jemiahw requested a review from Copilot June 4, 2025 00:58
Copilot

This comment was marked as outdated.

Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds handling for non-zero latency metrics in server-side tests to prevent intermittent failures by adjusting expected values to match actual latency (within a threshold).

  • Inserted fixLatencyMetric calls before metric assertions across read/write/query tests.
  • Added the fixLatencyMetric helper in query_test.go to clamp expected latency to actual values with a max check.
  • Corrected an assert.Equal argument order in bulk_write_test.go.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
api/single_entity_write_test.go Added fixLatencyMetric before metric comparisons.
api/single_entity_read_test.go Added fixLatencyMetric before metric comparisons.
api/query_test.go Added fixLatencyMetric calls and implemented the helper.
api/bulk_write_test.go Added fixLatencyMetric calls and fixed assert.Equal order.
Comments suppressed due to low confidence (2)

api/query_test.go:387

  • The helper comment is misleading: it says it replaces the latency metric in the actual metrics with the expected value, but the code actually replaces the expected value with the actual. Please update the comment to accurately describe this behavior.
// fixLatencyMetric is a helper function that fixes the latency metric in the actual metrics. Since latency is non-deterministic, it can cause

api/bulk_write_test.go:637

  • Swap the arguments to assert.Equal so the expected value comes first: assert.Equal(t, expectMetrics, server.metricsrec.Called).
assert.Equal(t, server.metricsrec.Called, expectMetrics)

daniel-nichter
daniel-nichter previously approved these changes Jun 4, 2025
qian-squareup
qian-squareup previously approved these changes Jun 4, 2025
@jemiahw jemiahw dismissed stale reviews from qian-squareup and daniel-nichter via ec563af June 4, 2025 17:44
@jemiahw jemiahw merged commit 092a3ed into master Jun 4, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants