diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/LICENSE b/instrumentation/opentelemetry-instrumentation-aiokafka/LICENSE new file mode 100644 index 0000000000..1ef7dad2c5 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright The OpenTelemetry Authors + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/README.rst b/instrumentation/opentelemetry-instrumentation-aiokafka/README.rst new file mode 100644 index 0000000000..dc4691c3d2 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/README.rst @@ -0,0 +1,22 @@ +OpenTelemetry aiokafka integration +====================================== + +|pypi| + +.. |pypi| image:: https://badge.fury.io/py/opentelemetry-instrumentation-aiokafka.svg + :target: https://pypi.org/project/opentelemetry-instrumentation-aiokafka/ + +Installation +------------ + +:: + + pip install opentelemetry-instrumentation-aiokafka + + +References +---------- + +* `OpenTelemetry aiokafka Instrumentation `_ +* `OpenTelemetry Project `_ +* `OpenTelemetry Python Examples `_ diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/pyproject.toml b/instrumentation/opentelemetry-instrumentation-aiokafka/pyproject.toml new file mode 100644 index 0000000000..0937cfee37 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/pyproject.toml @@ -0,0 +1,60 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "opentelemetry-instrumentation-aiokafka" +dynamic = ["version"] +description = "OpenTelemetry AIOKafka instrumentation" +readme = "README.rst" +license = "Apache-2.0" +requires-python = ">=3.7" +authors = [ + { name = "OpenTelemetry Authors", email = "cncf-opentelemetry-contributors@lists.cncf.io" }, +] +classifiers = [ + "Development Status :: 4 - Beta", + "Intended Audience :: Developers", + "License :: OSI Approved :: Apache Software License", + "Programming Language :: Python", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", +] +dependencies = [ + "opentelemetry-api ~= 1.5", + "opentelemetry-instrumentation == 0.43b0.dev", + "opentelemetry-semantic-conventions == 0.43b0.dev", +] + +[project.optional-dependencies] +instruments = [ + "aiokafka >= 0.8.0", +] +test = [ + "opentelemetry-instrumentation-aiokafka[instruments]", + "opentelemetry-test-utils == 0.43b0.dev", + "wrapt >= 1.0.0, < 2.0.0", + "mock; python_version<'3.8'", +] + +[project.entry-points.opentelemetry_instrumentor] +aiokafka = "opentelemetry.instrumentation.aiokafka:AIOKafkaInstrumentor" + +[project.urls] +Homepage = "https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-aiokafka" + +[tool.hatch.version] +path = "src/opentelemetry/instrumentation/aiokafka/version.py" + +[tool.hatch.build.targets.sdist] +include = [ + "/src", + "/tests", +] + +[tool.hatch.build.targets.wheel] +packages = ["src/opentelemetry"] diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/__init__.py b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/__init__.py new file mode 100644 index 0000000000..0187ec7707 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/__init__.py @@ -0,0 +1,128 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Instrument aiokafka to report instrumentation-aiokafka produced and consumed messages + +Usage +----- + +..code:: python + + from aiokafka import AIOKafkaConsumer, AIOKafkaProducer + from opentelemetry.instrumentation.aiokafka import AIOKafkaInstrumentor + + # Instrument aiokafka + AIOKafkaInstrumentor().instrument() + + async def send(): + # report a span of type producer with the default settings + producer = AIOKafkaProducer(bootstrap_servers=["localhost:9092"]) + await producer.send("my-topic", b"raw_bytes") + + async def consumer(): + # report a span of type consumer with the default settings + consumer = AIOKafkaConsumer("my-topic", group_id="my-group", bootstrap_servers=["localhost:9092"]) + async for message in consumer: + # process message + +The _instrument() method accepts the following keyword args: +tracer_provider (TracerProvider) - an optional tracer provider +produce_hook (Callable) - a function with extra user-defined logic to be performed before sending the message +this function signature is: +def produce_hook(span: Span, args, kwargs) +consume_hook (Callable) - a function with extra user-defined logic to be performed after consuming a message +this function signature is: +def consume_hook(span: Span, record: aiokafka.ConsumerRecord, args, kwargs) +for example: + +.. code: python + from aiokafka import AIOKafkaProducer + from opentelemetry.instrumentation.aiokafka import AIOKafkaInstrumentor + + def produce_hook(span, args, kwargs): + if span and span.is_recording(): + span.set_attribute("custom_user_attribute_from_produce_hook", "some-value") + def consume_hook(span, record, args, kwargs): + if span and span.is_recording(): + span.set_attribute("custom_user_attribute_from_consume_hook", "some-value") + + # instrument aiokafka with produce and consume hooks + AIOKafkaInstrumentor().instrument(produce_hook=produce_hook, consume_hook=consume_hook) + + # Using aiokafka as normal now will automatically generate spans, + # including user custom attributes added from the hooks + async def send(): + producer = AIOKafkaProducer(bootstrap_servers=["localhost:9092"]) + await producer.send("my-topic", b"raw_bytes") + +API +___ +""" +import typing + +import aiokafka # type: ignore[import-untyped] +from wrapt import wrap_function_wrapper # type: ignore[import-untyped] + +from opentelemetry import trace +from opentelemetry.instrumentation.aiokafka.package import _instruments +from opentelemetry.instrumentation.aiokafka.utils import ( + _wrap_getone, + _wrap_send, +) +from opentelemetry.instrumentation.aiokafka.version import __version__ +from opentelemetry.instrumentation.instrumentor import BaseInstrumentor +from opentelemetry.instrumentation.utils import unwrap + + +class AIOKafkaInstrumentor(BaseInstrumentor): + """An instrumentor for aiokafka module + See `BaseInstrumentor` + """ + + def instrumentation_dependencies(self) -> typing.Collection[str]: + return _instruments + + def _instrument(self, **kwargs: typing.Any) -> None: + """Instruments the aiokafka module + + Args: + **kwargs: Optional arguments + ``tracer_provider``: a TracerProvider, defaults to global. + ``produce_hook``: a callable to be executed just before producing a message + ``consume_hook``: a callable to be executed just after consuming a message + """ + tracer_provider = kwargs.get("tracer_provider") + produce_hook = kwargs.get("produce_hook") + consume_hook = kwargs.get("consume_hook") + + tracer = trace.get_tracer( + __name__, + __version__, + tracer_provider=tracer_provider, + schema_url="https://opentelemetry.io/schemas/1.11.0", + ) + + wrap_function_wrapper( + aiokafka.AIOKafkaProducer, "send", _wrap_send(tracer, produce_hook) + ) + wrap_function_wrapper( + aiokafka.AIOKafkaConsumer, + "getone", + _wrap_getone(tracer, consume_hook), + ) + + def _uninstrument(self, **kwargs: typing.Any) -> None: + unwrap(aiokafka.AIOKafkaProducer, "send") + unwrap(aiokafka.AIOKafkaConsumer, "getone") diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/package.py b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/package.py new file mode 100644 index 0000000000..12437a5a9d --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/package.py @@ -0,0 +1,16 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +_instruments = ("aiokafka >= 0.8.0",) diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/utils.py b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/utils.py new file mode 100644 index 0000000000..8f766eef35 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/utils.py @@ -0,0 +1,341 @@ +import json +import typing +from logging import getLogger + +import aiokafka # type: ignore[import-untyped] + +from opentelemetry import context, propagate, trace +from opentelemetry.context import Context +from opentelemetry.propagators import textmap +from opentelemetry.semconv.trace import ( + MessagingOperationValues, + SpanAttributes, +) +from opentelemetry.trace import Tracer +from opentelemetry.trace.span import Span + +_LOG = getLogger(__name__) + +ArgsT = typing.Tuple[typing.Any] +KwargsT = typing.Dict[str, typing.Any] +CarrierT = typing.List[typing.Tuple[str, bytes]] + + +class AIOKafkaPropertiesExtractor: + @staticmethod + def extract_bootstrap_servers( + instance: aiokafka.AIOKafkaClient, + ) -> typing.List[typing.Tuple[str, int]]: + return [(host, port) for (host, port, _) in instance.hosts] + + @staticmethod + def extract_client_id(instance: aiokafka.AIOKafkaClient) -> str: + return instance._client_id + + @staticmethod + def extract_get_group_id( + instance: aiokafka.AIOKafkaConsumer, + ) -> typing.Optional[str]: + return instance._group_id + + @staticmethod + def _extract_argument( + key: str, + position: int, + default_value: typing.Any, + args: ArgsT, + kwargs: KwargsT, + ) -> typing.Any: + if len(args) > position: + return args[position] + return kwargs.get(key, default_value) + + @staticmethod + def extract_send_topic(args: ArgsT, kwargs: KwargsT) -> str: + """extract topic from `send` method arguments in AIOKafkaProducer class""" + return AIOKafkaPropertiesExtractor._extract_argument( + "topic", 0, "unknown", args, kwargs + ) + + @staticmethod + def extract_send_value( + args: ArgsT, kwargs: KwargsT + ) -> typing.Optional[typing.Any]: + """extract value from `send` method arguments in AIOKafkaProducer class""" + return AIOKafkaPropertiesExtractor._extract_argument( + "value", 1, None, args, kwargs + ) + + @staticmethod + def extract_send_key( + args: ArgsT, kwargs: KwargsT + ) -> typing.Optional[typing.Any]: + """extract key from `send` method arguments in AIOKafkaProducer class""" + return AIOKafkaPropertiesExtractor._extract_argument( + "key", 2, None, args, kwargs + ) + + @staticmethod + def extract_send_headers( + args: ArgsT, kwargs: KwargsT + ) -> typing.Optional[CarrierT]: + """extract headers from `send` method arguments in AIOKafkaProducer class""" + return AIOKafkaPropertiesExtractor._extract_argument( + "headers", 5, None, args, kwargs + ) + + @staticmethod + def extract_send_partition( + instance: aiokafka.AIOKafkaProducer, args: ArgsT, kwargs: KwargsT + ) -> typing.Optional[int]: + """extract partition `send` method arguments, using the `_partition` method in AIOKafkaProducer class""" + try: + topic = AIOKafkaPropertiesExtractor.extract_send_topic( + args, kwargs + ) + key = AIOKafkaPropertiesExtractor.extract_send_key(args, kwargs) + value = AIOKafkaPropertiesExtractor.extract_send_value( + args, kwargs + ) + partition = AIOKafkaPropertiesExtractor._extract_argument( + "partition", 4, None, args, kwargs + ) + key_bytes, value_bytes = instance._serialize(topic, key, value) + valid_types = (bytes, bytearray, memoryview, type(None)) + if ( + type(key_bytes) not in valid_types + or type(value_bytes) not in valid_types + ): + return None + + return instance._partition( + topic, partition, key, value, key_bytes, value_bytes + ) + except Exception as exception: # pylint: disable=W0703 + _LOG.debug("Unable to extract partition: %s", exception) + return None + + +ProduceHookT = typing.Optional[typing.Callable[[Span, ArgsT, KwargsT], None]] +ConsumeHookT = typing.Optional[ + typing.Callable[[Span, aiokafka.ConsumerRecord, ArgsT, KwargsT], None] +] + + +class AIOKafkaContextGetter(textmap.Getter[CarrierT]): + def get( + self, carrier: CarrierT, key: str + ) -> typing.Optional[typing.List[str]]: + if carrier is None: + return None + + for item_key, value in carrier: + if item_key == key and value is not None: + return [value.decode()] + return None + + def keys(self, carrier: CarrierT) -> typing.List[str]: + if carrier is None: + return [] + return [key for (key, value) in carrier] + + +class AIOKafkaContextSetter(textmap.Setter[CarrierT]): + def set(self, carrier: CarrierT, key: str, value: str) -> None: + if carrier is None or key is None: + return + + if value: + carrier.append((key, value.encode())) + else: + carrier.append((key, b"")) + + +_kafka_getter: AIOKafkaContextGetter = AIOKafkaContextGetter() +_kafka_setter: AIOKafkaContextSetter = AIOKafkaContextSetter() + + +def _enrich_span( + span: Span, + bootstrap_servers: typing.List[typing.Tuple[str, int]], + group_id: typing.Optional[str], + client_id: str, + topic: str, + key: typing.Optional[str], + offset: typing.Optional[int], + partition: typing.Optional[int], + operation: MessagingOperationValues, +) -> None: + if not span.is_recording(): + return + + span.set_attribute(SpanAttributes.MESSAGING_SYSTEM, "kafka") + span.set_attribute( + SpanAttributes.MESSAGING_URL, json.dumps(bootstrap_servers) + ) + span.set_attribute(SpanAttributes.MESSAGING_CLIENT_ID, client_id) + + if group_id is not None: + span.set_attribute( + SpanAttributes.MESSAGING_KAFKA_CONSUMER_GROUP, group_id + ) + + span.set_attribute(SpanAttributes.MESSAGING_DESTINATION_NAME, topic) + span.set_attribute(SpanAttributes.MESSAGING_OPERATION, operation.value) + + if partition is not None: + span.set_attribute( + SpanAttributes.MESSAGING_KAFKA_DESTINATION_PARTITION, partition + ) + + if key is not None: + span.set_attribute(SpanAttributes.MESSAGING_KAFKA_MESSAGE_KEY, key) + + if offset is not None: + span.set_attribute( + SpanAttributes.MESSAGING_KAFKA_MESSAGE_OFFSET, offset + ) + + +def _get_span_name(operation: str, topic: str) -> str: + return f"{topic} {operation}" + + +def _wrap_send(tracer: Tracer, produce_hook: ProduceHookT) -> typing.Callable: + async def _traced_send( + func: typing.Callable[..., typing.Awaitable[None]], + instance: aiokafka.AIOKafkaProducer, + args: ArgsT, + kwargs: KwargsT, + ) -> None: + headers = AIOKafkaPropertiesExtractor.extract_send_headers( + args, kwargs + ) + if headers is None: + headers = [] + kwargs["headers"] = headers + + topic = AIOKafkaPropertiesExtractor.extract_send_topic(args, kwargs) + bootstrap_servers = ( + AIOKafkaPropertiesExtractor.extract_bootstrap_servers( + instance.client + ) + ) + client_id = AIOKafkaPropertiesExtractor.extract_client_id( + instance.client + ) + partition = AIOKafkaPropertiesExtractor.extract_send_partition( + instance, args, kwargs + ) + key = AIOKafkaPropertiesExtractor.extract_send_key(args, kwargs) + span_name = _get_span_name("send", topic) + with tracer.start_as_current_span( + span_name, kind=trace.SpanKind.PRODUCER + ) as span: + _enrich_span( + span=span, + bootstrap_servers=bootstrap_servers, + group_id=None, + client_id=client_id, + topic=topic, + key=key, + offset=None, + partition=partition, + operation=MessagingOperationValues.PUBLISH, + ) + propagate.inject( + headers, + context=trace.set_span_in_context(span), + setter=_kafka_setter, + ) + try: + if callable(produce_hook): + produce_hook(span, args, kwargs) + except Exception as hook_exception: # pylint: disable=W0703 + _LOG.exception(hook_exception) + + return await func(*args, **kwargs) + + return _traced_send + + +def _create_consumer_span( + tracer: Tracer, + consume_hook: ConsumeHookT, + record: aiokafka.ConsumerRecord, + extracted_context: Context, + bootstrap_servers: typing.List[typing.Tuple[str, int]], + group_id: typing.Optional[str], + client_id: str, + args: ArgsT, + kwargs: KwargsT, +) -> None: + span_name = _get_span_name("receive", record.topic) + with tracer.start_as_current_span( + span_name, + context=extracted_context, + kind=trace.SpanKind.CONSUMER, + ) as span: + new_context = trace.set_span_in_context(span, extracted_context) + token = context.attach(new_context) + _enrich_span( + span=span, + bootstrap_servers=bootstrap_servers, + group_id=group_id, + client_id=client_id, + topic=record.topic, + key=record.key, + offset=record.offset, + partition=record.partition, + operation=MessagingOperationValues.RECEIVE, + ) + try: + if callable(consume_hook): + consume_hook(span, record, args, kwargs) + except Exception as hook_exception: # pylint: disable=W0703 + _LOG.exception(hook_exception) + context.detach(token) + + +def _wrap_getone( + tracer: Tracer, + consume_hook: ConsumeHookT, +) -> typing.Callable: + async def _traced_getone( + func: typing.Callable[..., typing.Awaitable[aiokafka.ConsumerRecord]], + instance: aiokafka.AIOKafkaConsumer, + args: ArgsT, + kwargs: KwargsT, + ) -> aiokafka.ConsumerRecord: + record = await func(*args, **kwargs) + + if record: + bootstrap_servers = ( + AIOKafkaPropertiesExtractor.extract_bootstrap_servers( + instance._client + ) + ) + group_id = AIOKafkaPropertiesExtractor.extract_get_group_id( + instance + ) + client_id = AIOKafkaPropertiesExtractor.extract_client_id( + instance._client + ) + + extracted_context = propagate.extract( + record.headers, getter=_kafka_getter + ) + _create_consumer_span( + tracer, + consume_hook, + record, + extracted_context, + bootstrap_servers, + group_id, + client_id, + args, + kwargs, + ) + return record + + return _traced_getone diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/version.py b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/version.py new file mode 100644 index 0000000000..2e4aa8c751 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/src/opentelemetry/instrumentation/aiokafka/version.py @@ -0,0 +1,15 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +__version__ = "0.43b0.dev" diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/tests/__init__.py b/instrumentation/opentelemetry-instrumentation-aiokafka/tests/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/tests/test_instrumentation.py b/instrumentation/opentelemetry-instrumentation-aiokafka/tests/test_instrumentation.py new file mode 100644 index 0000000000..b870b70ac4 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/tests/test_instrumentation.py @@ -0,0 +1,40 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from unittest import TestCase + +import aiokafka +from wrapt import BoundFunctionWrapper + +from opentelemetry.instrumentation.aiokafka import AIOKafkaInstrumentor + + +class TestKafka(TestCase): + def test_instrument_api(self) -> None: + instrumentation = AIOKafkaInstrumentor() + + instrumentation.instrument() + self.assertTrue( + isinstance(aiokafka.AIOKafkaProducer.send, BoundFunctionWrapper) + ) + self.assertTrue( + isinstance(aiokafka.AIOKafkaConsumer.getone, BoundFunctionWrapper) + ) + + instrumentation.uninstrument() + self.assertFalse( + isinstance(aiokafka.AIOKafkaProducer.send, BoundFunctionWrapper) + ) + self.assertFalse( + isinstance(aiokafka.AIOKafkaConsumer.getone, BoundFunctionWrapper) + ) diff --git a/instrumentation/opentelemetry-instrumentation-aiokafka/tests/test_utils.py b/instrumentation/opentelemetry-instrumentation-aiokafka/tests/test_utils.py new file mode 100644 index 0000000000..25ee15178c --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-aiokafka/tests/test_utils.py @@ -0,0 +1,277 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# pylint: disable=unnecessary-dunder-call + +import asyncio +from unittest import TestCase, mock + +from opentelemetry.instrumentation.aiokafka.utils import ( + AIOKafkaPropertiesExtractor, + _create_consumer_span, + _get_span_name, + _kafka_getter, + _kafka_setter, + _wrap_getone, + _wrap_send, +) +from opentelemetry.semconv.trace import MessagingOperationValues +from opentelemetry.trace import SpanKind + +try: + from unittest.mock import AsyncMock +except ImportError: + # Fallback for python 3.7 + from mock import AsyncMock + + +class TestUtils(TestCase): + def setUp(self) -> None: + super().setUp() + self.topic_name = "test_topic" + self.args = [self.topic_name] + self.headers = [] + self.kwargs = {"partition": 0, "headers": self.headers} + + @mock.patch( + "opentelemetry.instrumentation.aiokafka.utils.AIOKafkaPropertiesExtractor.extract_bootstrap_servers" + ) + @mock.patch( + "opentelemetry.instrumentation.aiokafka.utils.AIOKafkaPropertiesExtractor.extract_send_partition" + ) + @mock.patch("opentelemetry.instrumentation.aiokafka.utils._enrich_span") + @mock.patch("opentelemetry.trace.set_span_in_context") + @mock.patch("opentelemetry.propagate.inject") + def test_wrap_send_with_topic_as_arg( + self, + inject: mock.MagicMock, + set_span_in_context: mock.MagicMock, + enrich_span: mock.MagicMock, + extract_send_partition: mock.MagicMock, + extract_bootstrap_servers: mock.MagicMock, + ) -> None: + self.wrap_send_helper( + inject, + set_span_in_context, + enrich_span, + extract_send_partition, + extract_bootstrap_servers, + ) + + @mock.patch( + "opentelemetry.instrumentation.aiokafka.utils.AIOKafkaPropertiesExtractor.extract_bootstrap_servers" + ) + @mock.patch( + "opentelemetry.instrumentation.aiokafka.utils.AIOKafkaPropertiesExtractor.extract_send_partition" + ) + @mock.patch("opentelemetry.instrumentation.aiokafka.utils._enrich_span") + @mock.patch("opentelemetry.trace.set_span_in_context") + @mock.patch("opentelemetry.propagate.inject") + def test_wrap_send_with_topic_as_kwarg( + self, + inject: mock.MagicMock, + set_span_in_context: mock.MagicMock, + enrich_span: mock.MagicMock, + extract_send_partition: mock.MagicMock, + extract_bootstrap_servers: mock.MagicMock, + ) -> None: + self.args = [] + self.kwargs["topic"] = self.topic_name + self.wrap_send_helper( + inject, + set_span_in_context, + enrich_span, + extract_send_partition, + extract_bootstrap_servers, + ) + + def wrap_send_helper( + self, + inject: mock.MagicMock, + set_span_in_context: mock.MagicMock, + enrich_span: mock.MagicMock, + extract_send_partition: mock.MagicMock, + extract_bootstrap_servers: mock.MagicMock, + ) -> None: + tracer = mock.MagicMock() + produce_hook = mock.MagicMock() + original_send_callback = AsyncMock() + kafka_producer = mock.MagicMock() + expected_span_name = _get_span_name("send", self.topic_name) + + wrapped_send = _wrap_send(tracer, produce_hook) + retval = asyncio.run( + wrapped_send( + original_send_callback, kafka_producer, self.args, self.kwargs + ) + ) + + extract_bootstrap_servers.assert_called_once_with( + kafka_producer.client + ) + extract_send_partition.assert_called_once_with( + kafka_producer, self.args, self.kwargs + ) + tracer.start_as_current_span.assert_called_once_with( + expected_span_name, kind=SpanKind.PRODUCER + ) + + span = tracer.start_as_current_span().__enter__.return_value + enrich_span.assert_called_once_with( + span=span, + group_id=None, + client_id=kafka_producer.client._client_id, + bootstrap_servers=extract_bootstrap_servers.return_value, + topic=self.topic_name, + key=None, + offset=None, + partition=extract_send_partition.return_value, + operation=MessagingOperationValues.PUBLISH, + ) + + set_span_in_context.assert_called_once_with(span) + context = set_span_in_context.return_value + inject.assert_called_once_with( + self.headers, context=context, setter=_kafka_setter + ) + + produce_hook.assert_called_once_with(span, self.args, self.kwargs) + + original_send_callback.assert_called_once_with( + *self.args, **self.kwargs + ) + self.assertEqual(retval, original_send_callback.return_value) + + @mock.patch("opentelemetry.propagate.extract") + @mock.patch( + "opentelemetry.instrumentation.aiokafka.utils._create_consumer_span" + ) + @mock.patch( + "opentelemetry.instrumentation.aiokafka.utils.AIOKafkaPropertiesExtractor.extract_bootstrap_servers" + ) + def test_wrap_next( + self, + extract_bootstrap_servers: mock.MagicMock, + _create_consumer_span: mock.MagicMock, + extract: mock.MagicMock, + ) -> None: + tracer = mock.MagicMock() + consume_hook = mock.MagicMock() + original_next_callback = AsyncMock() + kafka_consumer = mock.MagicMock() + + wrapped_next = _wrap_getone(tracer, consume_hook) + record = asyncio.run( + wrapped_next( + original_next_callback, kafka_consumer, self.args, self.kwargs + ) + ) + + extract_bootstrap_servers.assert_called_once_with( + kafka_consumer._client + ) + bootstrap_servers = extract_bootstrap_servers.return_value + + original_next_callback.assert_called_once_with( + *self.args, **self.kwargs + ) + self.assertEqual(record, original_next_callback.return_value) + + extract.assert_called_once_with(record.headers, getter=_kafka_getter) + context = extract.return_value + + _create_consumer_span.assert_called_once_with( + tracer, + consume_hook, + record, + context, + bootstrap_servers, + kafka_consumer._group_id, + kafka_consumer._client._client_id, + self.args, + self.kwargs, + ) + + @mock.patch("opentelemetry.trace.set_span_in_context") + @mock.patch("opentelemetry.context.attach") + @mock.patch("opentelemetry.instrumentation.aiokafka.utils._enrich_span") + @mock.patch("opentelemetry.context.detach") + def test_create_consumer_span( + self, + detach: mock.MagicMock, + enrich_span: mock.MagicMock, + attach: mock.MagicMock, + set_span_in_context: mock.MagicMock, + ) -> None: + tracer = mock.MagicMock() + consume_hook = mock.MagicMock() + bootstrap_servers = mock.MagicMock() + extracted_context = mock.MagicMock() + record = mock.MagicMock() + group_id = mock.MagicMock() + client_id = mock.MagicMock() + + _create_consumer_span( + tracer, + consume_hook, + record, + extracted_context, + bootstrap_servers, + group_id, + client_id, + self.args, + self.kwargs, + ) + + expected_span_name = _get_span_name("receive", record.topic) + + tracer.start_as_current_span.assert_called_once_with( + expected_span_name, + context=extracted_context, + kind=SpanKind.CONSUMER, + ) + span = tracer.start_as_current_span.return_value.__enter__() + set_span_in_context.assert_called_once_with(span, extracted_context) + attach.assert_called_once_with(set_span_in_context.return_value) + + enrich_span.assert_called_once_with( + span=span, + group_id=group_id, + client_id=client_id, + bootstrap_servers=bootstrap_servers, + topic=record.topic, + key=record.key, + offset=record.offset, + partition=record.partition, + operation=MessagingOperationValues.RECEIVE, + ) + consume_hook.assert_called_once_with( + span, record, self.args, self.kwargs + ) + detach.assert_called_once_with(attach.return_value) + + @mock.patch( + "opentelemetry.instrumentation.aiokafka.utils.AIOKafkaPropertiesExtractor" + ) + def test_kafka_properties_extractor( + self, + kafka_properties_extractor: mock.MagicMock, + ): + kafka_properties_extractor._serialize.return_value = (None, None) + kafka_properties_extractor._partition.return_value = "partition" + assert ( + AIOKafkaPropertiesExtractor.extract_send_partition( + kafka_properties_extractor, self.args, self.kwargs + ) + == "partition" + ) diff --git a/tox.ini b/tox.ini index e3d3a1fe16..d7dd3ec77e 100644 --- a/tox.ini +++ b/tox.ini @@ -228,6 +228,10 @@ envlist = py3{7,8,9,10,11}-test-instrumentation-aio-pika{7,8,9} pypy3-test-instrumentation-aio-pika{7,8,9} + ; opentelemetry-instrumentation-aiokafka + py3{7,8,9,10,11}-test-instrumentation-aiokafka + pypy3-test-instrumentation-aiokafka + ; opentelemetry-instrumentation-kafka-python py3{7,8,9,10,11}-test-instrumentation-kafka-python pypy3-test-instrumentation-kafka-python @@ -310,6 +314,7 @@ changedir = test-instrumentation-aio-pika: instrumentation/opentelemetry-instrumentation-aio-pika/tests test-instrumentation-aiohttp-client: instrumentation/opentelemetry-instrumentation-aiohttp-client/tests test-instrumentation-aiohttp-server: instrumentation/opentelemetry-instrumentation-aiohttp-server/tests + test-instrumentation-aiokafka: instrumentation/opentelemetry-instrumentation-aiokafka/tests test-instrumentation-aiopg: instrumentation/opentelemetry-instrumentation-aiopg/tests test-instrumentation-asgi: instrumentation/opentelemetry-instrumentation-asgi/tests test-instrumentation-asyncpg: instrumentation/opentelemetry-instrumentation-asyncpg/tests @@ -380,6 +385,8 @@ commands_pre = aio-pika{7,8,9}: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-aio-pika[test] + aiokafka: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-aiokafka[test] + kafka-python: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-kafka-python[test] confluent-kafka: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-confluent-kafka[test] @@ -549,6 +556,7 @@ commands_pre = python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-remoulade[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-fastapi[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-jinja2[test] + python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-aiokafka[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-kafka-python[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-confluent-kafka[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-logging[test] @@ -625,6 +633,7 @@ commands_pre = -e {toxinidir}/instrumentation/opentelemetry-instrumentation-asyncpg \ -e {toxinidir}/instrumentation/opentelemetry-instrumentation-celery \ -e {toxinidir}/instrumentation/opentelemetry-instrumentation-pika \ + -e {toxinidir}/instrumentation/opentelemetry-instrumentation-aiokafka \ -e {toxinidir}/instrumentation/opentelemetry-instrumentation-kafka-python \ -e {toxinidir}/instrumentation/opentelemetry-instrumentation-confluent-kafka \ -e {toxinidir}/instrumentation/opentelemetry-instrumentation-dbapi \