Repository navigation
Conversation
…ed with a driver class A handler argument annotated with a broker's own client class is treated as a message field, so the failure arrives once per message as a pydantic validation error about a missing field. Nothing in it points at the real cause, which is that the working annotation lives in faststream.<broker>.annotations under the same name as the driver class. Each broker's annotations module now registers the driver classes it wraps, and the shared call-model construction consults that registry when the handler's model is built. A hit raises SetupError naming the import to use instead.
borisalekseev
left a comment
There was a problem hiding this comment.
The tests are very implicit and know too much about the implementation details. Implement the tests through public behavior, checking pytest.raises with full help message matching when calling broker.start.
IvanKirpichnikov
left a comment
There was a problem hiding this comment.
Hi. I don’t like your solution because of the use of mutable variables.
Let’s do it this way.
Let’s move this function to a separate method, which we’ll override in each subscriber, and there we’ll perform the annotation check.
…a global registry Addresses review. The import-time registry is gone, along with the hook in the shared FastDepends config. SubscriberUsecase.__call__ hands its closure body to a new _create_call method. Each broker's root subscriber overrides it, checks the decorated handler against its own immutable table of driver classes, and delegates to super(). The tables sit in each broker's annotations module, beside the annotations they name. That also moves the failure to decoration time, which is what the issue asked for. Building the call model resolves the same hints later, so a handler whose annotations cannot be resolved is reported there as before rather than at decoration. Tests go through the public decorator and match the full message, one per broker in its misconfiguration module.
|
Reworked as asked. The registry and its import-time population are gone, along with the hook in @IvanKirpichnikov That also moves the failure to decoration time, which is what the issue title asked for and what I had argued against. You were right. Building the call model resolves the same hints later, so reading them at decoration adds no failure that would not have happened at startup anyway, and the @borisalekseev tests now go through the public decorator with the full message matched, one per broker in its misconfiguration module, plus one that the correct annotations are accepted. Nothing touches internals. Since the check is at decoration time, One behaviour change I did not gate. A broker built with mypy, ruff, codespell, bandit and semgrep are clean. pyright is unchanged against |
|
I think we are able to implement this logic internally. What about extending the broker configuration with a field called |
In my opinion, it’s a good idea. e.g However, in this case, you will have to abandon the module from which the required type hint needs to be imported. |
| # Building the call model resolves the same hints and reports the same | ||
| # failure, so decoration stays silent about it. | ||
| return |
There was a problem hiding this comment.
I didn’t understand what kind of error might occur.
Can you provide an example of code with an error?
There was a problem hiding this comment.
from __future__ import annotations
from typing import TYPE_CHECKING
from faststream.redis import RedisBroker
if TYPE_CHECKING:
from decimal import Decimal
broker = RedisBroker()
@broker.subscriber("channel")
async def handler(price: Decimal) -> None: ...Without the guard, get_type_hints fails while the module is still being imported:
NameError: name 'Decimal' is not defined
With it, that handler behaves exactly as it does on main, failing later with the message it already gives today:
pydantic.errors.PydanticUserError: `handler` is not fully defined; you should define `Decimal`,
then call `handler.model_rebuild()`.
So the guard is there to keep a handler that FastStream already rejects from being rejected earlier, and with a worse message. I have made the comment say that rather than leaving it vague.
There was a problem hiding this comment.
Let’s not ignore the error after all.
The error message contains at least one contradiction. It asks to call handler.model_rebuild(), although such a method doesn’t exist for handler.
…ceptionGroup Review request. Each offending argument becomes its own SetupError, and the group message names the handler. Note that an ExceptionGroup is not a SetupError, so anything catching SetupError around subscriber registration no longer sees this.
…nnotation-hints # Conflicts: # tests/brokers/redis/test_misconfigure.py
…nfig Addresses review. The table moves onto BrokerConfig as underlying_driver_annotations, so the check happens once in the original _create_call and the six per-broker overrides are gone. Each broker's config supplies its own table as the field default. Values are the context annotations themselves rather than an import path, so the message names the context key instead of a module and a name. A row whose annotation is a Depends rather than a Context has no key to name, and says only that FastStream provides one. The get_type_hints guard is gone as well. Removing it surfaced a real defect it had been hiding: a publisher decorator applied before the subscriber hands _create_call its HandlerCallWrapper, and resolving that object's hints reads the wrapper class's own annotations with an empty namespace. Unwrap to the original call first.
reflected |
IvanKirpichnikov
left a comment
There was a problem hiding this comment.
Why not put CONTEXT_ANNOTATIONS next to the broker’s config?
And I initially liked the idea of specifying the required import from faststream.redis.annotations import Redis, but now it’s specified as Annotated[Redis, Context("broker._connection")]. Can we bring this functionality back?
…me the import again Addresses review. CONTEXT_ANNOTATIONS moves into each broker's config module. Both sides of a row are import paths, so the table needs no imports of its own and the lazy import it used to need is gone. The six annotations modules are untouched by this branch again. The message names the import to use rather than reconstructing an Annotated form, which also restores it for the one row whose annotation is a Depends and had no context key to name.
There was a problem hiding this comment.
What do you think about doing it this way?
Let
underlying_driver_annotations: Mapping[Any, Any | UnderlyingDriverAnnotation]
@dataclass
class UnderlyingDriverAnnotation:
type_hint: Any
module: str
name: strThen you’ll get:
underlying_driver_annotations = {
aiopika.KafkaBroker: faststream.KafkaBroker,
redis.asyncio.Redis: UnderlyingDriverAnnotation(faststream.RedisBroker, faststream, "RedisBroker"),
}If UnderlyingDriverAnnotation is specified, the metadata for the error is taken from there: the module and the name, and we get
from faststream import RedisBroker
If a regular type hint is specified, there will be no import hint.
We also need to add the implementation of underlying_driver_annotations via the brokers’ __init__. For example
NatsBroker(..., underlying_driver_annotations={...})
I think that in this case, we will need to merge the default and custom ones.
Addresses review. The per-broker defaults become a dataclass field that each broker overrides with its own factory, rather than a _default_driver_annotations() hook, so the defaults sit on the config's public surface. The merge itself moves to a resolved_underlying_driver_annotations property. underlying_driver_annotations now keeps whatever the caller passed instead of being overwritten in __post_init__, and the resolved view is built per read. A custom row still adds to a broker's defaults rather than replacing them. The confluent and MQTT configs no longer chain into a base __post_init__, since the base no longer defines one.
Upstream moved broker_dependencies to Sequence, restyled the Redis params TypedDict from Annotated to docstrings, made include_routers variadic, and enabled PLC0415. The driver annotation rows keep their deferred imports, exempted in ruff.toml the way upstream exempts its own, since the objects only exist once the package is built. The bare driver generics in the Redis misconfigure tests are the mistake under test, so they take a type-arg ignore rather than a parameter that would change the asserted message.
IvanKirpichnikov
left a comment
There was a problem hiding this comment.
Hi. Overall, everything’s fine.
Let’s just change the function names from _context_annotations to _context_annotations_factory and do a git pull main.
Keeps both members added after BrokerConfig.extra_context: upstream's __post_init__, which tuple-ifies broker_dependencies, and this branch's resolved_underlying_driver_annotations property. Upstream tightened the subscriber dependencies parameter from Iterable to Sequence in the same range; _create_call is typed to match.
…_annotations_factory Review request: the module-level helper each broker config passes to default_factory reads as the annotations themselves rather than as the factory producing them.
|
Both done.
One thing worth flagging from the merge. Gates on the merged head: ruff, codespell, mypy, pyright, pyrefly, bandit, |
| "faststream/{confluent,kafka,nats,rabbit,redis}/configs/broker.py" = [ | ||
| "PLC0415", | ||
| ] | ||
|
|
||
| "faststream/mqtt/broker/config.py" = [ | ||
| "PLC0415", | ||
| ] | ||
|
|
There was a problem hiding this comment.
Let’s better explicitly set the noqa: PLC0415.
Lancetnik
left a comment
There was a problem hiding this comment.
Thanks for sticking with this through all the rounds. The error message is exactly what we want, and the table next to the broker config reads well. I think the check runs at the wrong step, though, and moving it removes most of the remaining problems.
Validate the built CallModel, not the raw signature
By setup time fast_depends has already built a CallModel for the handler, and it holds everything this check needs. Here is a handler run through build_call on main:
from __future__ import annotations
def dep(r: Redis) -> int: ...
@broker.subscriber("ch")
async def h(msg: str, direct: Redis, opt: Optional[Redis], ok: RedisCtx, d: int = Depends(dep)): ...params: msg: str, direct: Redis, opt: Optional[Redis]
flat_params: msg, direct, opt, r: Redis <- `r` comes from Depends
custom_fields: ['ok'] <- context annotations never reach params
dependencies: ['d']
Iterating dependent.flat_params in HandlerItem._setup, right after set_wrapped, fixes these:
- Arguments of
Dependsare caught. Todaydef dep(r: Redis)passes, because only the handler's own hints are read.flat_paramsalso includes the broker- and subscriber-leveldependencies. - No heuristic is needed. fast_depends already puts
Contextannotations incustom_fields, so a driver class found inflat_paramsis always one that would be validated as message data. We don't need the "Annotatedmeans OK" rule. - It fixes a regression.
get_type_hintsathints.py:40has no guard. A handler that imports a type underTYPE_CHECKINGwithfrom __future__ import annotationsnow fails at decoration with a bareNameError, and onmainit works.build_call_modelalready resolves the hints and reports failures the same way it does today. - Less code.
_create_call, reading_declared_call, and theunwraphandling for stacked decorators can all go. - The error comes from
broker.start(), which is where @borisalekseev asked for it.
The config side (resolved_underlying_driver_annotations, UnderlyingDriverAnnotation) can stay as it is. Only the thing being checked changes.
Required
- Move the check to the
CallModelas described above. - Unwrap
Optional[X]/X | None.field_typeholds the whole union, and the lookup is an exact match, soredis: Redis | None = Noneis not caught. That case is worse than the one in the issue: the handler silently getsNone. - Skip the check entirely when FastDepends is off (
apply_types=False). Nothing is injected in that mode, soContextand the suggested annotations don't work either, and the check has nothing to validate. Note thatbuild_call_modelruns before theuse_fastdependsbranch inFastDependsConfig.build_call, so a model always exists. The guard has to be explicit. - Tests:
- The same test is copied into six
test_misconfigure.pyfiles. Please move it into a shared base testcase undertests/brokers/base/, with each broker providing only its driver class and expected import. - Add cases for a driver class inside
Depends, forOptional, and forapply_types=False(that one must not raise). test_annotation_behind_depends_names_its_importhas noDependsin it. Please rename it or make it match its name.
- The same test is copied into six
Should fix
ruff.toml: replace the file-levelPLC0415ignores with an inline# noqa: PLC0415, as @IvanKirpichnikov asked.- Top-level export:
UnderlyingDriverAnnotationis exported from the top-levelfaststreampackage. Nobody asked for that export. Please keep it next to the configs and make itkw_only. - Typing: narrow
Mapping[Any, Any]on the new config fields and on the broker/router params, e.g. toMapping[Any, UnderlyingDriverAnnotation | Any]. - MQTT docstrings: the MQTT broker and router take
underlying_driver_annotationswithout a docstring entry. - Router tables: please check how the broker's and a router's tables combine with
include_router. I've only read the code, not run it.ConfigComposition.__getattr__returnsconfigs[0], so a router's own rows might apply only to subscribers declared beforeinclude_router, or might not merge with the broker's. Once the check reads the table at setup time, a test for this would settle it. - FastAPI integration: it builds its own
Dependantthroughget_dependent, not aCallModel. That integration is deprecated, so please don't handle it, just guard so the check doesn't break on it.
Minor
- One argument, one exception: a single bad argument still raises an
ExceptionGroup, soexcept SetupErrormisses it. A bareSetupErrorfor a single offender would be friendlier. - Stale description: the PR description still describes
MappingProxyTypeconstants in theannotationsmodules, six subscriber overrides, a try/except aroundget_type_hintsand a bareSetupError. None of that matches the code any more.
Upstream renamed BrokerConfigType to BrokerConfigType_co, so the configs package exports the new name beside this branch's UnderlyingDriverAnnotation. The same range turned on slotscheck's require-subclass (ag2ai#3220), which flags UnderlyingDriverAnnotation for having no slots. The next commit slots it.
…figs, keyword-only and slotted
|
All in except the MQTT docstrings (8), which need a decision from you, see the end. 1. The check reads the built It walks The 2. 3. 4. Tests. The six copies are one base testcase,
Against the source of the merge commit, before any of these fixes, every new case goes red. 5. 6. Export. Dropped from 7. Typing. 9. Router tables. You read it right. Once a router was included, 10. FastAPI. Covered by the guard in 3. The integration already builds its broker with Minor. A single bad argument raises a bare @IvanKirpichnikov the Gates, on the merge commit before any fix and on the new head: The slotscheck error is the 8. MQTT docstrings, not done.
|
|
and to pull main |
IvanKirpichnikov
left a comment
There was a problem hiding this comment.
And add documentation
| UnderlyingDriverAnnotations: TypeAlias = ( | ||
| Mapping[Any, UnderlyingDriverAnnotation | Any] | None | ||
| ) |
There was a problem hiding this comment.
I apologize, i meant to do it like this
| UnderlyingDriverAnnotations: TypeAlias = ( | |
| Mapping[Any, UnderlyingDriverAnnotation | Any] | None | |
| ) | |
| UnderlyingDriverAnnotations: TypeAlias = Mapping[Any, UnderlyingDriverAnnotation | Any] |
And just use
var: UnderlyingDriverAnnotations: = ...
var: UnderlyingDriverAnnotations | None = ...…d use it for every mapping
…ms and _find_mapped RedisBrokerParams now takes UnderlyingDriverAnnotations | None, like the other brokers' constructor parameter, and _find_mapped reads the alias instead of a bare Mapping.
|
@IvanKirpichnikov everything from your 10-05 review and the 10-07 follow-up is in, and
The four older threads that are still open are done too and only need resolving. The @Lancetnik item 3 reverses item 6 of your 09-23 review. I kept it because the new docs import One question from my 09-24 comment is still open. The MQTT broker and router are the only constructors without a docstring for the parameter, because D417 wants the whole Locally on f3f48c0, 4687 passed, 43 skipped, 7 xfailed on the non-connected suite, diff coverage is 95% (4 of 86 lines), and ruff, codespell, typos, mypy, pyright, pyrefly, bandit, slotscheck, import-linter, semgrep, |
Description
Annotating a handler argument with a broker's own client class produced a pydantic validation error about the message schema, once per message, with nothing in it pointing at the real cause.
Now the subscriber refuses it when it is set up, from
broker.start(), aTestBrokeror schema generation, and names the import that works:Fixes #3065
Changes
The check reads the handler's built
CallModel.SubscriberUsecase._build_fastdepends_modelcallscheck_context_annotations(faststream/_internal/endpoint/subscriber/hints.py) right afterHandlerItem._setup. fast_depends has already putContextannotations incustom_fieldsandDependsindependencies, so any driver class left in the model'sparamsis one that would be validated as message data. No heuristic aboutAnnotatedis needed, and the handler's hints are never resolved a second time.The walk covers the handler's own arguments and those of every dependency, including broker- and subscriber-level
dependencies. An argument found inside a dependency names it:Optional[X],X | NoneandAnnotated[X, ...]with plain metadata are unwrapped before the lookup. One bad argument raises a bareSetupError; several raise anExceptionGroupof them headed with the handler name.Nothing is checked when FastDepends is off (
apply_types=False), since nothing is injected then and the suggested annotations would not work either. The FastAPI integration builds its broker that way and supplies its own model throughget_dependent, so it is skipped too.The table lives on the broker config.
BrokerConfighas two fields,default_driver_annotations, which each broker's config fills from a_context_annotations_factorybeside it, andunderlying_driver_annotations, whatever the user passed.resolved_underlying_driver_annotationsmerges them with the user's rows winning. 30 rows ship, covering every entry in the issue's table plus the broker, message and producer classes theannotationsmodules already wrap. The factories import inside the function becausefaststream.<broker>.annotationsreaches the config module through the broker, and each of those imports carries# noqa: PLC0415.A row's value is an
UnderlyingDriverAnnotation(type_hint=..., module=..., name=...), a keyword-only, slotted dataclass infaststream._internal.configs, or a bare annotation. The value decides the message: anUnderlyingDriverAnnotationnames the import, a bare one says to use the context annotation FastStream provides.Users can add rows. Every broker and router takes
underlying_driver_annotations, merged over the shipped rows:ConfigCompositionmerges both tables across its configs the way it mergesextra_context, so a subscriber on an included router sees the broker's rows and the router's own, with the router's winning.Testing
tests/brokers/base/driver_annotations.pyis a base testcase inherited by every broker, which supplies only its driver class and expected import. It goes through the public decorator and the in-memoryTestBrokerand matches the full message. Cases: a plain driver class,X | None, plainAnnotatedandAnnotated | None, a driver class insideDepends, a handler on a router joined withinclude_router, the context annotation accepted,apply_types=Falsenot checked, and aTYPE_CHECKING-only hint accepted.tests/brokers/redis/test_misconfigure.pyholds the table behaviour that is not per broker: custom rows with and without an import, custom rows not replacing the defaults, a router's rows merging with the broker's, and several bad arguments reported together.The 68 added tests are the whole difference.
Diff coverage is measured without the connected jobs, which need brokers that are not available here, so CI should read it no lower.
connectedtests were not run.Static analysis, all clean:
Docker is not available here, so the
justrecipes were not used; these are the commands they wrap, run against the pinned versions.Notes for reviewers
The
MQTTBrokerandMQTTRouterconstructors have no docstring, and ruff's D417 rejects anArgs:section that leaves parameters out, sounderlying_driver_annotationsis undocumented there. Documenting it means documenting all 37 and 10 parameters, which I would rather do separately.underlying_driver_annotationsis not in redis'sNON_CONNECTION_PARAMS, so it also reaches redis-py's connection kwargs, which tolerate it. That is a one-line fix I am holding for a follow-up rather than growing this PR.faststream/asgi/annotations.pyandfaststream/opentelemetry/annotations.pyshipContextannotations too and have the same failure mode. Left out to keep this to the brokers.Two corrections to the issue's table. The NATS object store annotation is
faststream.nats.annotations.ObjectStorage, notObjectStore, and the MQTT client iszmqtt.client.MQTTClient. The rows use the real names.Type of change
Checklist
getting-started/context.mdabout the name collision if you want one.