Implement base object for handling triggers with basic data types

[pragnagopa]

[1st1]

Context

The original motivation for this feature is to allow creating functions for binding types that are not yet fully supported by the worker. The current behaviour is to simply reject them to be loaded with an error.

The proposed solution is to:

• add common base datatypes for trigger bindings and regular bindings;
• expose the underlying gRPC TypedData raw data like the nodejs worker.

Pros:

• This would mean that even if there's no direct implementation of a binding type, users would still be able to write functions that use it.

Cons:

• The main problem here is that by doing that, we would inadvertently expose Python Functions users to lots of low-level protocol details. In the short term, users will be able to use new bindings/azure services before they are officially supported by the Python worker.

• This essentially makes the low-level encoding—TypedData gRPC structs—an API. This complicates things for binding types and azure services, as they would have to maintain backwards compatibility w.r.t. how exactly bindings are encoded on the wire.

• Moreover, we'll still have to document how exactly those bindings are encoded on the wire, otherwise users won't be able to work with these new binding types.

Therefore, implementing this feature doesn't really make it easier for users to work with unsupported bindings. At best this would allow them to quickly prototype a function before the binding/integration is fully supported, but by doing so, they would be writing code that relies on private undocumented data formats.

---

Initial attempt to implement the feature

Invariants:

• We don't want to expose the actual gRPC protocol to the user.
• We don't want to mirror the TypedData gRPC message to equivalent Python classes—that's not Pythonic.
• We don't want to assume that all bindings are always encoded as JSON. There might be a new binding in the future that uses TypedData(int=..) gRPC type for encoding its values.

Therefore we want a high-level facade types for the TypedData. I've prototyped the following ABCs:

class RawInputData(abc.ABC):

    @abc.abstractmethod
    def get_str(self) -> typing.Optional[str]:
        pass

    @abc.abstractmethod
    def get_bytes(self) -> typing.Optional[bytes]:
        pass

    @abc.abstractmethod
    def get_int(self) -> typing.Optional[int]:
        pass

    @abc.abstractmethod
    def get_float(self) -> typing.Optional[float]:
        pass

    @abc.abstractmethod
    def get_json(self) -> typing.Any:
        pass

    # XXX: Not complete! Probably need to add support for TypedData(http)


class RawOutputData(RawInputData):

    @abc.abstractmethod
    def set_str(self, val: str):
        pass

    @abc.abstractmethod
    def set_bytes(self, val: bytes):
        pass

    @abc.abstractmethod
    def set_int(self, val: int):
        pass

    @abc.abstractmethod
    def set_float(self, val: float):
        pass

    @abc.abstractmethod
    def set_json(self, val: typing.Any):
        pass


class GenericBinding(abc.ABC):

    @property
    @abc.abstractmethod
    def raw_data(self) -> RawOutputData:
        pass


class GenericTrigger(abc.ABC):

    @property
    @abc.abstractmethod
    def raw_data(self) -> RawInputData:
        pass

    @property
    @abc.abstractmethod
    def raw_metadata(self) -> typing.Mapping[str, RawInputData]:
        pass

The idea is to inherit most of binding and trigger types from either GenericBinding or GenericTrigger classes. We would then rewrite every type implementation to use the raw_data and raw_metadata to ensure that any modifications to the raw data are always synchronized.

The problem is that this increases the overall complexity of the code. That will result in documentation becoming more complex too.

---

Question

So the real question here: do we really want this? Maybe the correct approach is to simply add official support of a binding/trigger/service before documenting it and letting users use it?

After all we made it easy to implement new bindings:

azure-functions-python-worker/azure/worker/bindings/timer.py

Lines 10 to 36 in 9ed3f8a

	class TimerRequest(azf_abc.TimerRequest):
	def __init__(self, *, past_due: bool) -> None:
	self.__past_due = past_due
	@property
	def past_due(self):
	return self.__past_due
	class TimerRequestConverter(meta.InConverter,
	binding='timerTrigger', trigger=True):
	@classmethod
	def check_python_type(cls, pytype: type) -> bool:
	return issubclass(pytype, azf_abc.TimerRequest)
	@classmethod
	def from_proto(cls, data: protos.TypedData, *,
	pytype: typing.Optional[type],
	trigger_metadata) -> typing.Any:
	if data.WhichOneof('data') != 'json':
	raise NotImplementedError
	info = json.loads(data.json)
	return TimerRequest(
	past_due=info.get('IsPastDue', False))

/cc @asavaritayal @christopheranderson @brettcannon @elprans

[brettcannon]

Could you expose the gRPC TypedData with a underscore-leading setup so that those who want to poke at it can, but that it isn't an officially supported part of the API?

[1st1]

Yes, we could do that. I'm still not sure why that would be useful though:

• IMO, exposing raw gRPC messages is only possible for "in" bindings.

  Exposing them for "out" bindings is outright dangerous: the Azure Functions Web Host can be easily crashed in case of a malformed message. Same as with CPython, we allow users to create "broken" code objects, but it's impossible to fully protect the interpreter from segfaulting in all cases.

  Therefore, if we do this only for "in" bindings, then users will still need to wait for us adding support for new bindings/services officially.

• Another problem with gRPC is that its datastructures are not pleasant to work with. Without looking at the protobuf spec, it would be hard for users to make sense of the data (I'm speaking from my own experience here). And even if somebody is persistent enough to extract some useful data from raw gRPC messages, it can stop working the other day when some minor change to the platform is made.

• It seems that we don't expose raw data for C# functions: all bindings are exposed via nice and usable C# types. It looks like NodeJS worker is the exception here, and I feel that Python should follow the C# path.

[brettcannon]

@1st1 neither do I, but you seemed to want to expose something and I was trying to minimize the work to do that. 😉

[1st1]

Yeah :)

So if @asavaritayal and @christopheranderson are OK with this, I'd prefer to close this issue and simply implement the remaining bindings properly, without exposing any of gRPC low-level messaging to the end-users.

[1st1]

As per our conf call with @asavaritayal, @pragnagopa, and @paulbatum, we decided to postpone the final decision on this issue. We'll continue implementing high-level rich Python types for bindings for now. We'll reconsider adding a generic type that exposes raw JSON data based on user experience and our own experience implementing rich types.

[asavaritayal]

Quick update on this issue -

Per feedback over the last couple of months, the high-level rich Python types are much easier to use than the basic types we have for node. Since the azure.functions package is now published to PyPI, rich types simplify development via type hints, intelli-sense, etc. making it easy to orchestrate data from the underlying binding sources.

Closing this issue for now. It can be re-opened if hear otherwise. /cc @paulbatum

[asavaritayal]

Resurrecting this thread per discussion with @anthonychu.

Is it possible that while we support rich types for first class bindings such as HTTP, Blob, Event Grid, etc. , we still allow users to bring their own binding (per our extensibility model) and use it as a basic data type object in their function code? We've got a few new bindings spring up lately - Signal R, Event Hubs, etc. and it seems that Python will start to lose out on scenarios because we're not constantly building support for each of these bindings in the language worker.

[pragnagopa]

cc: @maiqbal11 @paulbatum

We should look into adding support for using context object similar to javascript functions. Exposing context object does not expose TypedData defined in the protobuf which is an internal implementation detail.

We should allow following function signatures:

1. Without importing azure.functions, code should be able to work directly with context object and accessing input data via context.bindingData

def main(context, msg) -> str:

2. Giving an option to the customer to use azure.functions package,

def main(context, msg: azf.QueueMessage) -> str:

[asavaritayal]

We should look into adding support for using context object similar to javascript functions. Exposing context object does not expose TypedData defined in the protobuf which is an internal implementation detail.

Open to comments on this but I think we should keep the function signature consistent for the end user regardless of whether the binding is of a "basic" or "rich" type.

For e.g.

if req is of type azure.functions.HttpRequest type but msg is of type str

def main(req: azure.functions.HttpRequest, msg: str):

    name = req.params.get('name')
    logging.info(f'property from rich type: {name}')

    logging.info(f'basic data type: {msg}')