test(pubsub): fix flakiness and formatting issues in schemas acceptance tests#34026
Open
torreypayne wants to merge 3 commits into
Open
test(pubsub): fix flakiness and formatting issues in schemas acceptance tests#34026torreypayne wants to merge 3 commits into
torreypayne wants to merge 3 commits into
Conversation
quartzmo
reviewed
Jun 8, 2026
| @subscription = subscription_admin.create_subscription name: pubsub.subscription_path(random_subscription_id), | ||
| topic: @topic.name, | ||
| ack_deadline_seconds: 60 | ||
| sleep 5 |
Member
There was a problem hiding this comment.
I wonder if instead of using a hard-coded sleep (guaranteed to add 5s to the test suite), we could use expect_with_retry from google-cloud-ruby/google-cloud-pubsub/samples/acceptance/helper.rb instead? It's used in other tests for this gem. Maybe something like:
# pubsub_subscribe_avro_records
expect_with_retry "pubsub_subscribe_avro_records" do
# Publish the message inside the retry loop
publisher = pubsub.publisher @topic.name
publisher.publish record.to_json
expected = /.../
assert_output expected do
subscribe_avro_records subscription_id: @subscription.name, avsc_file: nil
end
end
Member
Author
There was a problem hiding this comment.
ya I don't like the sleep 5 solution at all and I've been digging into an alternative fix; let me try that out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #34025
This PR fixes flakiness and formatting issues in the schemas_test.rb acceptance tests for google-cloud-pubsub samples. These issues were causing CI failures, particularly on newer Ruby versions (3.3+).
Root Causes & Fixes
1. Ruby 3.3 Hash Formatting Difference:
Issue: Avro subscriber tests used assert_output with exact string matching. The expected output had spaces around => (e.g., {"name" => "Alaska"}), but Ruby 3.3 represents Hashes without spaces (e.g., {"name"=>"Alaska"}), causing consistent failures on attempt 1.
Fix: Updated the assertions to use space-tolerant regular expressions (e.g., /Received a binary-encoded message:\n{"name"\s*=>\s*"Alaska",\s*"post_abbr"\s*=>\s*"AK"}/).
2. Subscription Propagation Delay (Flakiness):
Issue: Messages were being published immediately after subscription creation. Because Pub/Sub subscription creation is eventually consistent, this occasionally led to messages being lost if they were published before the subscription was active in all zones.
Fix: Added sleep 5 after subscription creation in the Avro tests, matching the behavior already present in the Proto tests.
3. Out-of-Order Message Delivery:
Issue: The revisions test published two messages (one for Rev A, one for Rev B) and asserted their receipt using a single regex that enforced order (Alaska then California). Since Pub/Sub delivery is unordered by default, they could arrive in reverse order, causing the assertion to fail.
Fix: Changed the assertion to use capture_io and separate assert_match calls for each expected message, making the test order-independent.
4. Shadowed Errors in expect_with_retry:
Note: The test helper expect_with_retry retries the test block up to 5 times. If the first attempt failed (due to the issues above) but acknowledged the messages, subsequent attempts would receive nothing and fail with actual: "". Minitest then reported the last failure, shadowing the actual root cause of the initial failure. Fixing the root causes above prevents this retry-failure loop.
Verification Results
Verified locally on fix-acceptance-tests-flakiness branch (which is based on main and only contains these fixes):
GOOGLE_APPLICATION_CREDENTIALS=... GOOGLE_CLOUD_PROJECT=helical-zone-771 bundle exec ruby acceptance/schemas_test.rbOutput:
Finished in 392.020107s, 0.0332 runs/s, 0.1377 assertions/s.
13 runs, 54 assertions, 0 failures, 0 errors, 0 skips
All existing tests passed successfully on the first attempt.