-
Notifications
You must be signed in to change notification settings - Fork 10
Fix intermittent test failure #91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
A number of Etre tests verify the server side metrics after performing some operation (create, update, delete, etc.). The existing assertions assume latency will be 0. Usually it is, but occasionaly on the GitHub servers running automated tests on the PR, the latency will be a few milliseconds. This causes the test to fail since the latency metric will have a value that doesn't match the 0 in the assertion. To fix this, I've added a method "fixLatencyMetric". This method finds any latency metrics in the "expect" list and updates the value to match the corresponding metric from the "actual" list. To make sure the latency metric value is not garbage, fixLatencyMetrics also takes a max latency and asserts that the actual latency is between 0 and max. This way we still check the latency metric has some reasonable value, but we won't fail the test if the latency is a few milliseconds instead of 0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds handling for non-zero latency metrics in server-side tests to prevent intermittent failures by adjusting expected values to match actual latency (within a threshold).
- Inserted
fixLatencyMetric
calls before metric assertions across read/write/query tests. - Added the
fixLatencyMetric
helper inquery_test.go
to clamp expected latency to actual values with a max check. - Corrected an
assert.Equal
argument order inbulk_write_test.go
.
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
File | Description |
---|---|
api/single_entity_write_test.go | Added fixLatencyMetric before metric comparisons. |
api/single_entity_read_test.go | Added fixLatencyMetric before metric comparisons. |
api/query_test.go | Added fixLatencyMetric calls and implemented the helper. |
api/bulk_write_test.go | Added fixLatencyMetric calls and fixed assert.Equal order. |
Comments suppressed due to low confidence (2)
api/query_test.go:387
- The helper comment is misleading: it says it replaces the latency metric in the actual metrics with the expected value, but the code actually replaces the expected value with the actual. Please update the comment to accurately describe this behavior.
// fixLatencyMetric is a helper function that fixes the latency metric in the actual metrics. Since latency is non-deterministic, it can cause
api/bulk_write_test.go:637
- Swap the arguments to
assert.Equal
so the expected value comes first:assert.Equal(t, expectMetrics, server.metricsrec.Called)
.
assert.Equal(t, server.metricsrec.Called, expectMetrics)
ec563af
A number of Etre tests verify the server side metrics after performing some operation (create, update, delete, etc.). The existing assertions assume latency will be 0. Usually it is, but occasionaly on the GitHub servers running automated tests on the PR, the latency will be a few milliseconds. This causes the test to fail since the latency metric will have a value that doesn't match the 0 in the assertion.
To fix this, I've added a method "fixLatencyMetric". This method finds any latency metrics in the "expect" list and updates the value to match the corresponding metric from the "actual" list. To make sure the latency metric value is not garbage, fixLatencyMetrics also takes a max latency and asserts that the actual latency is between 0 and max. This way we still check the latency metric has some reasonable value, but we won't fail the test if the latency is a few milliseconds instead of 0.