-
Notifications
You must be signed in to change notification settings - Fork 46
feat: Enable sqs -> lambda support for DSM #604
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
purple4reina
merged 19 commits into
DataDog:main
from
michael-zhao459:michael.zhao/lambda-support
Jun 5, 2025
+217
−0
Merged
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
ac374c2
set context for sqs-> lambda
michael-zhao459 db0c56c
needed queue name for consistent hash
michael-zhao459 c12e2dc
add context prop test
michael-zhao459 8f4f2d8
comment
michael-zhao459 e7ebe21
only import when DSM is enabled
michael-zhao459 67efeca
fix lint
michael-zhao459 2c94f07
fix
michael-zhao459 701ed35
fix
michael-zhao459 dd26482
refactorings
michael-zhao459 116b7d9
fix
michael-zhao459 6c8ec7f
Update datadog_lambda/dsm.py
michael-zhao459 9dbf2de
fix
michael-zhao459 779e6a5
fixes
michael-zhao459 e6a8b4e
fixes
michael-zhao459 7507357
test fixes
michael-zhao459 bbf219b
fix
michael-zhao459 254b864
fix
michael-zhao459 dda744e
unit test fixes
michael-zhao459 1f888b1
fixes
michael-zhao459 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
add context prop test
- Loading branch information
commit c12e2dc9612bafe3b23b0c228e9ece151c9ddef4
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -563,6 +563,171 @@ def return_type_test(event, context): | |
self.assertEqual(result, test_result) | ||
self.assertFalse(MockPrintExc.called) | ||
|
||
@patch.dict(os.environ, {"DD_DATA_STREAMS_ENABLED": "true"}) | ||
def test_datadog_lambda_wrapper_dsm_sqs_context_pathway_verification(self): | ||
with patch( | ||
"ddtrace.internal.datastreams.processor.get_connection" | ||
) as mock_get_connection: | ||
|
||
mock_conn = unittest.mock.MagicMock() | ||
mock_response = unittest.mock.MagicMock() | ||
mock_response.status = 200 | ||
mock_conn.getresponse.return_value = mock_response | ||
mock_get_connection.return_value = mock_conn | ||
|
||
def updated_get_datastreams_context(message): | ||
""" | ||
Updated version that handles the correct message formats | ||
""" | ||
import base64 | ||
import json | ||
|
||
context_json = None | ||
message_body = message | ||
try: | ||
body = message.get("Body") | ||
if body: | ||
message_body = json.loads(body) | ||
except (ValueError, TypeError): | ||
pass | ||
|
||
message_attributes = message_body.get( | ||
"MessageAttributes" | ||
) or message_body.get("messageAttributes") | ||
if not message_attributes: | ||
return None | ||
|
||
if "_datadog" not in message_attributes: | ||
return None | ||
|
||
datadog_attr = message_attributes["_datadog"] | ||
|
||
if message_body.get("Type") == "Notification": | ||
if datadog_attr.get("Type") == "Binary": | ||
context_json = json.loads( | ||
base64.b64decode(datadog_attr["Value"]).decode() | ||
) | ||
elif "StringValue" in datadog_attr: | ||
context_json = json.loads(datadog_attr["StringValue"]) | ||
elif "stringValue" in datadog_attr: | ||
context_json = json.loads(datadog_attr["stringValue"]) | ||
elif "BinaryValue" in datadog_attr: | ||
context_json = json.loads(datadog_attr["BinaryValue"].decode()) | ||
else: | ||
print(f"DEBUG: Unhandled datadog_attr format: {datadog_attr}") | ||
|
||
return context_json | ||
|
||
with patch( | ||
"ddtrace.internal.datastreams.botocore.get_datastreams_context", | ||
updated_get_datastreams_context, | ||
): | ||
|
||
# Step 1: Create a message with some context in the message attributes | ||
|
||
from ddtrace.internal.datastreams.processor import DataStreamsProcessor | ||
|
||
processor_instance = DataStreamsProcessor() | ||
|
||
with patch( | ||
"ddtrace.internal.datastreams.processor.DataStreamsProcessor", | ||
return_value=processor_instance, | ||
): | ||
|
||
parent_ctx = processor_instance.new_pathway() | ||
|
||
parent_ctx.set_checkpoint( | ||
["direction:out", "topic:upstream-topic", "type:sqs"], | ||
now_sec=1640995200.0, | ||
payload_size=512, | ||
) | ||
parent_hash = parent_ctx.hash | ||
encoded_parent_context = parent_ctx.encode_b64() | ||
|
||
sqs_event = { | ||
"Records": [ | ||
{ | ||
"eventSource": "aws:sqs", | ||
"eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:test", | ||
"Body": "test message body", | ||
"messageAttributes": { | ||
"_datadog": { | ||
"stringValue": json.dumps( | ||
{ | ||
"dd-pathway-ctx-base64": encoded_parent_context | ||
} | ||
) | ||
} | ||
}, | ||
} | ||
] | ||
} | ||
|
||
# Step 2: Call the handler | ||
@wrapper.datadog_lambda_wrapper | ||
def lambda_handler(event, context): | ||
return {"statusCode": 200, "body": "processed"} | ||
|
||
result = lambda_handler(sqs_event, get_mock_context()) | ||
self.assertEqual(result["statusCode"], 200) | ||
|
||
# New context set after handler call | ||
current_ctx = processor_instance._current_context.value | ||
self.assertIsNotNone( | ||
current_ctx, | ||
"Data streams context should be set after processing SQS message", | ||
) | ||
|
||
# Step 3: Check that hash in this context is the child of the hash you passed | ||
# Step 4: Check that the right checkpoint was produced during call to handler | ||
|
||
found_sqs_checkpoint = False | ||
for bucket_time, bucket in processor_instance._buckets.items(): | ||
for aggr_key, stats in bucket.pathway_stats.items(): | ||
edge_tags_str, hash_value, parent_hash_recorded = aggr_key | ||
edge_tags = edge_tags_str.split(",") | ||
|
||
if ( | ||
"direction:in" in edge_tags | ||
and "topic:test" in edge_tags | ||
and "type:sqs" in edge_tags | ||
): | ||
found_sqs_checkpoint = True | ||
|
||
# EXPLICIT PARENT-CHILD HASH RELATIONSHIP TEST | ||
self.assertEqual( | ||
parent_hash_recorded, | ||
parent_hash, | ||
f"Parent hash must be preserved: " | ||
f"expected {parent_hash}, got {parent_hash_recorded}", | ||
) | ||
self.assertEqual( | ||
hash_value, | ||
current_ctx.hash, | ||
f"Child hash must match current context: " | ||
f"expected {current_ctx.hash}, got {hash_value}", | ||
) | ||
self.assertNotEqual( | ||
hash_value, | ||
parent_hash_recorded, | ||
f"Child hash ({hash_value}) must be different from " | ||
f"parent hash ({parent_hash_recorded}) - proves parent-child", | ||
) | ||
self.assertGreaterEqual( | ||
stats.payload_size.count, | ||
1, | ||
"Should have one payload size measurement", | ||
) | ||
|
||
break | ||
|
||
self.assertTrue( | ||
found_sqs_checkpoint, | ||
"Should have found SQS consumption checkpoint in processor stats", | ||
) | ||
|
||
processor_instance.shutdown(timeout=0.1) | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These look great! Super easy to read and follow. |
||
|
||
class TestLambdaDecoratorSettings(unittest.TestCase): | ||
def test_some_envs_should_depend_on_dd_tracing_enabled(self): | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can vastly simplify these tests. This might warrant another pairing session, but I'll let you take a stab at it on your own first. Feel free to schedule something with me if you wanna go over it together.
There are two different files which you changed, and therefore two different test files that will need updating:
tests/test_wrapper.py
and a new filetests/test_dsm.py
.test_wrapper.py
For the
test_wrapper.py
file, we simply need to test that, based on the env varsDD_DATA_STREAMS_ENABLED
andDD_TRACE_ENABLED
we either do or do not callset_dsm_context
with the proper args. We'll push all of the verification of what happens inside ofset_dsm_context
to thetest_dsm.py
file.I'm a bit fan of
pytest
, which isn't yet imported and used in this file, which allows you to reuse the same code over and over again to create "parametrized" tests. You can accomplish this same thing usingunittest
(as this file already uses), though it would mean creating 4 different test methods.test_dsm.py
In the
test_dsm.py
file, this is where you'll assert to make sure that theset_dsm_context
works as expected. You'll want to include several tests:SQS
will do nothingThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lemme take a stab at these today, I will definitely schedule a meeting if I get stuck on any of these parts