Skip to content

Fix WQX3 array parameter serialization - #390

Merged
thodson-usgs merged 3 commits into
DOI-USGS:mainfrom
thodson-usgs:fix/wqx3-array-parameters-389
Aug 26, 2026
Merged

thodson-usgs merged 3 commits into
DOI-USGS:mainfrom
thodson-usgs:fix/wqx3-array-parameters-389

Conversation

@thodson-usgs

Copy link
Copy Markdown
Collaborator

Summary

  • preserve iterable WQX3 query values so httpx encodes them as repeated keys
  • retain semicolon-delimited serialization for legacy WQP endpoints
  • document and test array parameters for WQX3 result and station searches

Validation

  • pytest -q tests/wqp_test.py (34 passed)
  • coverage run -m pytest tests/ && coverage report -m (997 passed, 12 deselected; 98% coverage)
  • ruff check .
  • ruff format --check .
  • mypy
  • xenon --max-absolute C --max-modules B --max-average A dataretrieval
  • complexipy dataretrieval
  • lint-imports

Fixes #389

thodson-usgs and others added 2 commits August 25, 2026 10:17
The array-parameter fix derived the WQX3-vs-legacy decision a second time
at each call site to pick a delimiter, in opposite polarity from the branch
that had already picked the URL. Set both together instead, so the
"wqx3_url implies repeated params" invariant reads off one branch per arm.

_what drops use_wqx3 and its duplicate services_wqx3 membership test for a
flat three-way branch; the stack depth to _legacy_only_url is unchanged, so
the hand-tuned stacklevel in _warn_wqx3_unavailable still lands.

_query_with_retry now builds a new payload rather than writing joined
values back into the caller's dict. The legacy path previously mutated the
caller's kwargs -- storing None for scalars -- while the new WQX3 path did
not, so WQP_Metadata saw different parameters depending on the interface.

Live WQX3 checks confirm semicolon lists are wrong for every array
parameter tested, not just siteType: huc, organization, providers, siteid,
countycode, statecode, sampleMedia, characteristicType, characteristicName,
pCode and countrycode all return correct unions only with repeated keys.
The tests' "River/Stream" example is a WQX3 Location_Type response value,
not a valid siteType filter term -- the live service rejects it -- so the
examples now use Stream/Well.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thodson-usgs
thodson-usgs marked this pull request as ready for review August 25, 2026 18:51
@thodson-usgs
thodson-usgs requested a review from ehinman August 25, 2026 18:51
@thodson-usgs

Copy link
Copy Markdown
Collaborator Author

@ehinman, revise as you see fit and merge. I don't need to review this one.

@ehinman ehinman left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change the term "iterable" to "list". Also, the user doesn't need to know how lists of parameters are handled in WQX3 vs legacy. Otherwise, code runs as expected.

Comment thread dataretrieval/wqp.py
Comment thread dataretrieval/wqp.py Outdated
Eight-digit hydrologic unit (HUC), delimited by semicolons.
huc : string or iterable of strings
Eight-digit hydrologic unit (HUC). Iterable values are encoded as
repeated parameters for WQX3 and semicolon-delimited for legacy WQP.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This last sentence isn't needed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

Comment thread dataretrieval/wqp.py
Radial-search distance in decimal miles.
pCode : string
Five-digit USGS parameter code, delimited by semicolons.
pCode : string or iterable of strings

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pCode : string or iterable of strings
pCode : string or list of strings

Comment thread dataretrieval/wqp.py Outdated
Five-digit USGS parameter code, delimited by semicolons.
pCode : string or iterable of strings
Five-digit USGS parameter code. Iterable values are encoded as
repeated parameters for WQX3 and semicolon-delimited for legacy WQP.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last sentence isn't needed.

Comment thread dataretrieval/wqp.py
characteristicName : string
One or more case-sensitive characteristic names, separated by
semicolons (https://www.waterqualitydata.us/public_srsnames/).
characteristicName : string or iterable of strings

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
characteristicName : string or iterable of strings
characteristicName : string or list of strings

Comment thread dataretrieval/wqp.py Outdated
Materialize non-string iterables before WQP serialization so repeated WQX3 parameters work for generators and Series while legacy requests retain semicolon encoding. Preserve reusable values in response metadata and keep the public parameter docs transport-neutral.
@thodson-usgs

Copy link
Copy Markdown
Collaborator Author

@ehinman, I accepted your suggestions. The only change was to make the implementation accept iterables, making the doc correct as written.

@thodson-usgs
thodson-usgs merged commit 7043932 into DOI-USGS:main Aug 26, 2026
11 checks passed
@thodson-usgs
thodson-usgs deleted the fix/wqx3-array-parameters-389 branch August 26, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WQX3 array parameters are incorrectly serialized with semicolons

2 participants