This content originally appeared on DEV Community and was authored by Murat K Ozcan
In our integration tests, Pact, a consumer-driven contract testing tool, has been instrumental in verifying interactions between services. However, when integrating Kafka and message queues, we encountered a perplexing issue where tests were cross-executing between different provider-consumer pairs. In this blog post, we'll delve into the problem we faced, why it was critical to resolve, and how we ultimately solved it.
You can find the solution in this PR.
The Problem: Cross-Execution of Tests Between Provider-Consumer Pairs
We had two repositories representing two distinct provider-consumer pairs:
Where The 2 repos have 2 unique contracts between them:
- MoviesAPI ↔ WebConsumer (HTTP-based interaction)
- MoviesAPI-event-producer ↔ WebConsumer-event-consumer (Kafka message-based interaction)
Each pair had its own Pact contracts and verification tests. Locally, everything worked flawlessly, in CI PRs as well.
However, when the consumer triggered a webhook in the provider repository (as part of the CI/CD pipeline), we encountered a strange issue:
- The Kafka test (
provider-kafka.pacttest.ts), intended for theMoviesAPI-event-producerandWebConsumer-event-consumer, was inadvertently executing tests meant for the HTTP-based pair (MoviesAPIandWebConsumer). - This cross-execution led to test failures and confusion, especially since the tests were running in the context of the wrong provider-consumer pair.
Root Cause Analysis
The crux of the problem was the handling of the PACT_PAYLOAD_URL environment variable:
The PACT_PAYLOAD_URL was being set globally, causing both test suites to pick it up, regardless of whether it was relevant to them.
This is the way things have to be, because during a webhook trigger, which is caused by a repository_dispatch, we want an HTTP request from the consumer repo to the provider repo instructing GitHub to start the webhook action.
A repository_dispatch in GitHub is basically an HTTP request to your GitHub project instructing GitHub to start any action or webhook. In our example, a repository_dispatch with the event type contract_requiring_verification_published will trigger the workflow.
In our local tests and in PRs, PACT_BROKER_BASE_URL is used, and during a web hook trigger it is replaced with PACT_PAYLOAD_URL. This is because we want to verify the newly published contract from the consumer repo PR that caused the trigger.
Recall the consumer and provider flow.
The key is that, when there are multiple repos, the provider has to run
test:provider-ci(#3)after the consumer runspublish:pact(#2)but before the consumer can runcan:i:deploy:consumer(#4). The trigger to runtest:provider-ci(#3)has to happen automatically, webhooks handle this.# Consumer npm run test:consumer # (1) npm run publish:pact # (2) npm run can:i:deploy:consumer # (4) # only on main npm run record:consumer:deployment --env=dev # (5) change the env param as needed # Provider npm run test:provider-ci # (3) triggered by webhooks npm run can:i:deploy:provider # (4) # only on main npm run record:provider:deployment --env=dev # (5) change the env param as needed
Why Was the Problem Worth Solving
Ensuring that each test suite only executes its intended tests is crucial for several reasons:
- Accuracy: Tests should only verify the contracts relevant to their specific provider-consumer pair.
- Reliability: Cross-execution can lead to false negatives, causing CI/CD pipelines to fail unnecessarily.
- Scalability: As more provider-consumer pairs are added, the problem would compound, leading to more significant issues down the line.
By resolving this issue, we aimed to improve the reliability of our test suites, streamline our CI/CD processes, and set a foundation for scalable contract testing.
How We Solved It
To address the issue, we undertook a systematic approach:
1. Refactoring the URL Handling Logic
We focused on the handlePactBrokerUrlAndSelectors function, which was responsible for configuring the verifier options based on the presence of a PACT_PAYLOAD_URL.
Original Issues:
- Complexity: The function had a high cyclomatic complexity, making it hard to read and maintain.
- Global Variable Usage: It relied on globally set environment variables, leading to unintended side effects.
-
Lack of Conditional Handling: There was no check to ensure that the
PACT_PAYLOAD_URLmatched the expected provider and consumer before using it.
Refactored Solution:
We broke down the handlePactBrokerUrlAndSelectors function into smaller, focused functions:
-
parseProviderAndConsumerFromUrl: Extracts the provider and consumer names from thePACT_PAYLOAD_URL. -
processPactPayloadUrl: Determines whether thePACT_PAYLOAD_URLshould be used based on whether it matches the expected provider and consumer. -
usePactPayloadUrlConfigures the verifier options to use thePACT_PAYLOAD_URLfor verification. -
usePactBrokerUrlAndSelectors: Configures the verifier options to use the Pact Broker URL and appropriate consumer version selectors.
By modularizing the code, we reduced complexity and made it easier to understand and maintain.
2. Implementing Conditional Logic
We introduced logic to ensure that a test suite only uses the PACT_PAYLOAD_URL if it matches its specific provider and consumer:
if (providerMatches && consumerMatches) {
usePactPayloadUrl(pactPayloadUrl, options);
return true; // Indicate that the Pact payload URL was used
} else {
console.log(
`PACT_PAYLOAD_URL does not match the provider (${options.provider}) and consumer (${consumer || 'all'}), ignoring it`
);
}
This check prevents a test suite from inadvertently using a PACT_PAYLOAD_URL intended for a different provider-consumer pair.
3. Ensuring Proper Fallback
If the PACT_PAYLOAD_URL is not provided or doesn't match, the verifier options are configured to use the Pact Broker URL and consumer version selectors:
// If pactPayloadUrl is not provided or doesn't match, use the Pact Broker URL and selectors
usePactBrokerUrlAndSelectors(
pactBrokerUrl,
consumer,
includeMainAndDeployed,
options
);
This ensures that the test suite can still run and verify contracts appropriately, even in the absence of a matching PACT_PAYLOAD_URL.
4. Updating Test Files
In our test files (provider-contract.pacttest.ts and provider-kafka.pacttest.ts), we made sure to specify the correct provider and consumer when building the verifier options. This information is crucial for the processPactPayloadUrl function to correctly determine whether to use the PACT_PAYLOAD_URL.
const options = buildVerifierOptions({
provider: 'MoviesAPI',
consumer: 'WebConsumer',
// ... other options
})
///
const options = buildMessageVerifierOptions({
provider: 'MoviesAPI-event-producer',
consumer: 'WebConsumer-event-consumer',
})
5. Verifying the Solution
We tested the refactored code in various scenarios:
-
Matching
PACT_PAYLOAD_URL: Ensured that the test suite used the payload URL when it matched the expected provider and consumer. -
Non-matching
PACT_PAYLOAD_URL: Confirmed that the test suite ignored the payload URL and used the Pact Broker URL and selectors instead. -
No
PACT_PAYLOAD_URL: Verified that the test suite defaulted to using the Pact Broker URL and selectors when the payload URL was not provided.
Conclusion
By refactoring our code and introducing conditional logic to handle the PACT_PAYLOAD_URL appropriately, we resolved the cross-execution issue in our Pact tests involving Kafka and message queues. This solution not only fixed the immediate problem but also enhanced the maintainability and scalability of our testing framework.
Key Takeaways:
- Modular Code Improves Maintainability: Breaking down complex functions into smaller, focused ones makes the codebase easier to understand and work with.
- Conditional Logic Prevents Cross-Execution: Implementing checks to ensure that only the intended tests are executed prevents cross-execution issues.
- Testing Various Scenarios: Verifying the solution under different conditions ensures robustness and reliability.
Final Thoughts:
Addressing this issue was crucial for maintaining the integrity of our testing process. As microservices architectures continue to grow in complexity, having reliable and maintainable testing practices becomes ever more important. We hope that sharing our experience will help others facing similar challenges in their testing journeys.
This content originally appeared on DEV Community and was authored by Murat K Ozcan
Murat K Ozcan | Sciencx (2024-10-16T14:56:26+00:00) Solving Cross-Execution Issues in Pact Testing with Kafka and Message Queues. Retrieved from https://www.scien.cx/2024/10/16/solving-cross-execution-issues-in-pact-testing-with-kafka-and-message-queues/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.