docs: Add comprehensive Mock Server testing documentation #798
Conversation
- Created new TESTING_CONNECTORS.md guide covering HttpMocker utilities - Included quick start examples and core component documentation - Added testing patterns for full refresh, incremental, pagination, and error handling - Documented advanced techniques like request builders and time freezing - Included real-world examples from Salesforce and Hubspot connectors - Added best practices and troubleshooting sections Co-Authored-By: AJ Steers <aj@airbyte.io>
Original prompt from AJ Steers |
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. Testing This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1760642702-expand-mock-server-testing-docs#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1760642702-expand-mock-server-testing-docsHelpful ResourcesPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
📝 WalkthroughWalkthroughA comprehensive testing guide for Airbyte connectors has been added, documenting mock HTTP utilities, request/response definitions, and testing patterns for streams, pagination, authentication, and error handling. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes A quick thought: Since this documentation establishes testing patterns for the CDK, would it be helpful to consider adding a reference link to this guide in the main Airbyte documentation or connector developer quickstart (if one exists)? That way, new contributors can easily discover these best practices, wdyt? Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive documentation for testing Airbyte connectors using the CDK's HttpMocker utilities. The documentation expands on the existing brief Mock Server section in CONTRIBUTING.md by providing detailed guidance for connector developers.
Key changes:
- Created a complete testing guide with examples ranging from basic to advanced patterns
- Documented core components (HttpMocker, HttpRequest, HttpResponse, response builders) with practical code examples
- Provided real-world examples adapted from production connectors (Salesforce and Hubspot)
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| ## Additional Resources | ||
|
|
||
| - [CDK Test Utilities API Reference](../airbyte_cdk/test/) | ||
| - [Example Tests in CDK](../../unit_tests/sources/mock_server_tests/) |
There was a problem hiding this comment.
The relative paths in these links may be incorrect. The first link uses ../airbyte_cdk/test/ which assumes this doc is in a subdirectory of the CDK root, but the second link uses ../../unit_tests/ which suggests a different directory structure. Verify these paths work correctly from the docs/ directory.
| - [Example Tests in CDK](../../unit_tests/sources/mock_server_tests/) | |
| - [Example Tests in CDK](../unit_tests/sources/mock_server_tests/) |
| # Second page | ||
| http_mocker.get( | ||
| HttpRequest("https://api.example.com/users", query_params={"page": "page2"}), | ||
| HttpResponse('{"users": [{"id": 2}], "next_page": null}', 200) |
There was a problem hiding this comment.
The JSON contains null which is not valid JSON. In JSON, null values should be written as null without quotes, but this appears to be inside a string literal. It should be either \"next_page\": null (without quotes around null) or the string should use proper JSON escaping.
| self._http_mocker.get( | ||
| HttpRequest( | ||
| "https://instance.salesforce.com/services/data/v57.0/queryAll", | ||
| query_params={"q": f"SELECT * FROM Account WHERE SystemModstamp >= {cursor_value}"} |
There was a problem hiding this comment.
The f-string formatting will not work correctly for datetime objects in SOQL queries. Salesforce SOQL requires specific datetime formatting (ISO 8601). The cursor_value should be formatted using .isoformat() or similar method to ensure proper SOQL syntax.
| query_params={"q": f"SELECT * FROM Account WHERE SystemModstamp >= {cursor_value}"} | |
| query_params={"q": f"SELECT * FROM Account WHERE SystemModstamp >= {cursor_value.isoformat()}"} |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
docs/TESTING_CONNECTORS.md (2)
203-216: Double-checkHttpMockerpagination example structureThe pagination snippet calls
read(source, config={}, catalog=catalog)without definingsourceorcatalog, and it never sets up the mock responses to match aCatalogBuilder. Could we either show their setup inline or mention that they come from the quick-start example to avoid confusing readers, wdyt?
382-451: Validate real-world example against actual buildersIn the Salesforce/Hubspot sections, the helper classes (
ConfigBuilder,ContactsResponseBuilder, etc.) look custom. Could we add a note that these builders come from their respective connector test suites (with paths) or swap in simplified inline builders so readers don’t hunt for missing definitions, wdyt?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
docs/TESTING_CONNECTORS.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
docs/TESTING_CONNECTORS.md
[grammar] ~83-~83: There might be a mistake here.
Context: ... the mocked response to return. You can specify body (string or bytes), status code, an...
(QB_NEW_EN)
[grammar] ~477-~477: There might be a mistake here.
Context: ...doesn't match any mocked request. Check: - URL matches exactly (including protocol)...
(QB_NEW_EN)
[grammar] ~489-~489: There might be a mistake here.
Context: .... ### Request Not Being Mocked Ensure: - You're using the @HttpMocker() decorat...
(QB_NEW_EN)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Manifest Server Docker Image Build
- GitHub Check: SDM Docker Image Build
- GitHub Check: Pytest (Fast)
| @HttpMocker() | ||
| def test_retry_on_rate_limit(http_mocker): | ||
| # Return rate limit error, then success | ||
| http_mocker.get( | ||
| HttpRequest("https://api.example.com/users"), | ||
| [ | ||
| HttpResponse('{"error": "Rate limit exceeded"}', 429), | ||
| HttpResponse('{"users": [{"id": 1}]}', 200) | ||
| ] | ||
| ) | ||
|
|
||
| output = read(source, config={}, catalog=catalog) | ||
| assert len(output.records) == 1 | ||
| ``` |
There was a problem hiding this comment.
Clarify retry example response list usage
In the rate-limit example, http_mocker.get(..., [HttpResponse(...), HttpResponse(...)]) assumes the mocker accepts a list of responses. From HttpMocker's current API, the overload expects response_sequence=[...] as a keyword argument. To keep the example runnable, could we switch to http_mocker.get(request, response_sequence=[...]), wdyt?
🤖 Prompt for AI Agents
In docs/TESTING_CONNECTORS.md around lines 224 to 237 the rate-limit example
passes a list of HttpResponse objects positionally to http_mocker.get, but
HttpMocker expects the list via the keyword response_sequence; update the
example to call http_mocker.get(HttpRequest(...),
response_sequence=[HttpResponse(...), HttpResponse(...)]) so the mocker uses the
intended response sequence and the snippet is runnable.
| @HttpMocker() | ||
| def test_request_count(http_mocker): | ||
| request = HttpRequest("https://api.example.com/users") | ||
| http_mocker.get(request, HttpResponse('{"users": []}', 200)) | ||
|
|
||
| # Perform sync | ||
| read(source, config={}, catalog=catalog) | ||
|
|
||
| # Assert request was called exactly once | ||
| http_mocker.assert_number_of_calls(request, 1) |
There was a problem hiding this comment.
Verify assert_number_of_calls availability
http_mocker.assert_number_of_calls isn’t part of the public API in HttpMocker today (the helper lives on the matcher registry). Should we update the snippet to use the supported http_mocker.assert_called_once(request) helper instead—or add a short note if we’re introducing a new helper, wdyt?
🤖 Prompt for AI Agents
In docs/TESTING_CONNECTORS.md around lines 344 to 353, the example calls
http_mocker.assert_number_of_calls which is not part of the public HttpMocker
API; replace that call with the supported helper
http_mocker.assert_called_once(request) in the snippet (or if you intend to add
a new public helper, add a short note above the snippet documenting the new API
and its behavior), ensuring the example uses only public HttpMocker methods and
the text reflects whichever approach you choose.
| from airbyte_cdk.models import AirbyteStreamStatus | ||
|
|
||
| @HttpMocker() | ||
| def test_stream_status(http_mocker): | ||
| http_mocker.get(request, response) | ||
|
|
||
| output = read(source, config={}, catalog=catalog) | ||
| statuses = output.get_stream_statuses("users") | ||
|
|
||
| assert statuses[0] == AirbyteStreamStatus.STARTED | ||
| assert statuses[1] == AirbyteStreamStatus.RUNNING | ||
| assert statuses[2] == AirbyteStreamStatus.COMPLETE |
There was a problem hiding this comment.
Confirm stream status helper exists
output.get_stream_statuses("users") doesn’t exist on the object returned by read (it’s a ReadOutput dataclass with records, logs, states). Maybe we can show how to filter output.logs for AirbyteLogMessage status entries instead, wdyt?
🤖 Prompt for AI Agents
In docs/TESTING_CONNECTORS.md around lines 361 to 372, the example calls
output.get_stream_statuses("users") which doesn't exist on the ReadOutput
dataclass; instead, filter output.logs for AirbyteMessage/AirbyteLogMessage
entries whose type is STREAM_STATUS and then extract the stream name and stream
status values for the "users" stream. Update the example to iterate over
output.logs, select only stream-status log messages, map them to their status
enums (AirbyteStreamStatus) and assert those values in order for the "users"
stream.
|
|
||
| - [CDK Test Utilities API Reference](../airbyte_cdk/test/) | ||
| - [Example Tests in CDK](../../unit_tests/sources/mock_server_tests/) | ||
| - [Salesforce Connector Tests](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-salesforce/unit_tests/integration/) | ||
| - [Hubspot Connector Tests](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-hubspot/unit_tests/integrations/) |
There was a problem hiding this comment.
Fix broken relative link for unit-test API reference
The link to ../airbyte_cdk/test/ 404s from docs/. Pointing directly to the rendered API docs (or the GitHub directory) would keep navigation smooth. Could we update it to https://github.com/airbytehq/airbyte/tree/master/airbyte-cdk/python/airbyte_cdk/test or similar, wdyt?
🤖 Prompt for AI Agents
In docs/TESTING_CONNECTORS.md around lines 495 to 499 the relative link
"../airbyte_cdk/test/" 404s from the docs site; replace that relative path with
a working absolute link such as
"https://github.com/airbytehq/airbyte/tree/master/airbyte-cdk/python/airbyte_cdk/test"
(or the rendered API docs URL) so the unit-test API reference points to the
correct GitHub directory or published docs.
docs: Add comprehensive Mock Server testing documentation
Summary
Created a new comprehensive guide (
docs/TESTING_CONNECTORS.md) for testing Airbyte connectors using the CDK'sHttpMockerutilities. The documentation was requested to expand on the existing brief Mock Server section in CONTRIBUTING.md.Key additions:
The documentation references existing test examples in the CDK repo and production connectors to provide practical, realistic guidance for connector developers.
Review & Testing Checklist for Human
Test Plan
response_builder.pyAPINotes
MySource())Session Info:
Summary by CodeRabbit