Skip to content

feat(logger): add logger buffer feature #6060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Mar 7, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
d1a7a58
First commit - Adding Buffer and Config classes
leandrodamascena Feb 9, 2025
6c4ef9c
First commit - Adding Buffer and Config classes
leandrodamascena Feb 9, 2025
e2dd0c4
First commit - Adding Buffer and Config classes
leandrodamascena Feb 9, 2025
511f147
First commit - Adding nox tests
leandrodamascena Feb 9, 2025
ebf49db
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 9, 2025
d24a258
Adding new parameter to the constructor
leandrodamascena Feb 9, 2025
0893e14
Starting tests
leandrodamascena Feb 10, 2025
505eb98
Adding initial logic + test
leandrodamascena Feb 13, 2025
dbfd708
Adding initial logic + test
leandrodamascena Feb 13, 2025
8a4e7bf
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 13, 2025
93f4a13
Adding initial logic + test
leandrodamascena Feb 13, 2025
35acff5
Unit tests for log comparasion
leandrodamascena Feb 13, 2025
ed3902d
Fixing logic to handle log flush
leandrodamascena Feb 13, 2025
e39229a
Start flushing logs
leandrodamascena Feb 13, 2025
27b1ce9
Adding more logic
leandrodamascena Feb 13, 2025
7b06ac9
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 17, 2025
69eda82
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 17, 2025
10d81d9
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 19, 2025
a98ee0a
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 20, 2025
58e26ae
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 21, 2025
e97d065
Adding more logic
leandrodamascena Feb 21, 2025
8b30fe5
Adding more logic
leandrodamascena Feb 21, 2025
5406a77
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 24, 2025
1893216
Refactoring buffer class
leandrodamascena Feb 24, 2025
ce3804e
Refactoring buffer class
leandrodamascena Feb 24, 2025
7f49bb4
Refactoring buffer class
leandrodamascena Feb 25, 2025
351d856
More refactor
leandrodamascena Feb 28, 2025
f7eeb2e
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 28, 2025
7d98138
Adding more coverage + documentation
leandrodamascena Feb 28, 2025
0c8cb3b
Adding decorator
leandrodamascena Feb 28, 2025
c977904
Refactoring variables name
leandrodamascena Mar 1, 2025
c9adbee
Adding more tests + propagating buffer configuration
leandrodamascena Mar 1, 2025
699a3d8
Adding exception fields
leandrodamascena Mar 2, 2025
49f308b
Adding e2e tests
leandrodamascena Mar 2, 2025
fa5c7f1
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 2, 2025
b1d81b5
sys.getframe is more performatic
leandrodamascena Mar 2, 2025
feaea2b
Chaging the max size
leandrodamascena Mar 3, 2025
a46e2ff
Removing unecessary files
leandrodamascena Mar 3, 2025
9368459
Fix tests
leandrodamascena Mar 3, 2025
1167850
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 5, 2025
f9db4cd
Flushing logs when log line is bigger than buffer size
leandrodamascena Mar 5, 2025
f30eed0
Flushing logs when log line is bigger than buffer size
leandrodamascena Mar 5, 2025
452cf91
Adding documentation
leandrodamascena Mar 5, 2025
2d8eb39
Adding documentation
leandrodamascena Mar 5, 2025
46a9ab9
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 5, 2025
a5a12a5
Adding documentation
leandrodamascena Mar 6, 2025
46fc1df
Addressing Andrea's feedback
leandrodamascena Mar 6, 2025
b1e2f99
Addressing Andrea's feedback
leandrodamascena Mar 6, 2025
4717cdc
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 6, 2025
f5618ea
Addressing Andrea's feedback
leandrodamascena Mar 6, 2025
1da9bf2
Refactoring parameters name + child logger + tests
leandrodamascena Mar 6, 2025
a64df93
Improve documentation
leandrodamascena Mar 6, 2025
078b33c
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 6, 2025
bd7084f
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 7, 2025
5c9eb79
Improve docstring
leandrodamascena Mar 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Adding more coverage + documentation
  • Loading branch information
leandrodamascena committed Feb 28, 2025
commit 7d9813844e084546a7220c1a367d29afd0af18a7
3 changes: 2 additions & 1 deletion aws_lambda_powertools/logging/buffer/cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,9 @@ def add(self, key: str, item: Any) -> None:
item_size = len(str(item))
if item_size > self.max_size_bytes:
warnings.warn(
message=f"Cannot add item to the buffer "
f"Item size {item_size} bytes exceeds total cache size {self.max_size_bytes} bytes",
PowertoolsUserWarning,
category=PowertoolsUserWarning,
stacklevel=2,
)
return
Expand Down
8 changes: 0 additions & 8 deletions aws_lambda_powertools/logging/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,3 @@ class OrphanedChildLoggerError(Exception):
"""

pass


class InvalidBufferItem(Exception):
"""
Raised when a buffer item exceeds the maximum allowed buffer size.
"""

pass
255 changes: 124 additions & 131 deletions aws_lambda_powertools/logging/logger.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
LOGGER_ATTRIBUTE_PRECONFIGURED,
)
from aws_lambda_powertools.logging.exceptions import (
InvalidBufferItem,
InvalidLoggerSamplingRateError,
OrphanedChildLoggerError,
)
Expand Down Expand Up @@ -510,136 +509,6 @@ def decorate(event, context, *args, **kwargs):

return decorate

def _create_and_flush_log_record(self, log_line: dict) -> None:
"""
Create and immediately flush a log record to the configured logger.

Parameters
----------
log_line : dict[str, Any]
Dictionary containing log record details with keys:
- 'level': Logging level
- 'filename': Source filename
- 'line': Line number
- 'msg': Log message
- 'function': Source function name
- 'extra': Additional context
- 'timestamp': Original log creation time

Notes
-----
Bypasses standard logging flow by directly creating and handling a log record.
Preserves original timestamp and source information.
"""
record = self._logger.makeRecord(
name=self.name,
level=log_line["level"],
fn=log_line["filename"],
lno=log_line["line"],
msg=log_line["msg"],
args=(),
exc_info=None,
func=log_line["function"],
extra=log_line["extra"],
)
record.created = log_line["timestamp"]
self._logger.handle(record)

def _add_log_record_to_buffer(
self,
level: int,
msg: object,
args: object,
exc_info: logging._ExcInfoType = None,
stack_info: bool = False,
extra: Mapping[str, object] | None = None,
) -> None:
"""
Add log record to buffer with intelligent tracer ID handling.

Parameters
----------
level : int
Logging level of the record.
msg : object
Log message to be recorded.
args : object
Additional arguments for the log message.
exc_info : logging._ExcInfoType, optional
Exception information for the log record.
stack_info : bool, optional
Whether to include stack information.
extra : Mapping[str, object], optional
Additional contextual information for the log record.

Raises
------
InvalidBufferItem
If the log record cannot be added to the buffer.

Notes
-----
Handles special first invocation buffering and migration of log records
between different tracer contexts.
"""
# Determine tracer ID, defaulting to first invoke marker
tracer_id = get_tracer_id()

try:
if tracer_id:
log_record: dict[str, Any] = _create_buffer_record(level=level, msg=msg, args=args, extra=extra)
self._buffer_cache.add(tracer_id, log_record)
except InvalidBufferItem as exc:
# Wrap and re-raise buffer addition error as warning
warnings.warn(
message=f"Cannot add item to the buffer: {str(exc)}",
category=PowertoolsUserWarning,
stacklevel=3,
)

def flush_buffer(self) -> None:
"""
Flush all buffered log records associated with current execution.

Notes
-----
Retrieves log records for current trace from buffer
Immediately processes and logs each record
Warning if some cache was evicted in that execution
Clears buffer after complete processing

Raises
------
Any exceptions from underlying logging or buffer mechanisms
will be propagated to caller
"""
tracer_id = get_tracer_id()

# Flushing log without a tracer id? Return
if not tracer_id:
return

# is buffer empty? return
buffer = self._buffer_cache.get(tracer_id)
if not buffer:
return

# Process log records
for log_line in buffer:
self._create_and_flush_log_record(log_line)

# Has items evicted?
if self._buffer_cache.has_items_evicted(tracer_id):
warnings.warn(
message="Some logs are not displayed because they were evicted from the buffer. "
"Increase buffer size to store more logs in the buffer",
category=PowertoolsUserWarning,
stacklevel=2,
)

# Clear the entire cache
self._buffer_cache.clear()

def debug(
self,
msg: object,
Expand Down Expand Up @@ -1139,6 +1008,130 @@ def _determine_log_level(self, level: str | int | None) -> str | int:
# Powertools log level is set, we use this
return powertools_log_level.upper()

# FUNCTIONS for Buffering log

def _create_and_flush_log_record(self, log_line: dict) -> None:
"""
Create and immediately flush a log record to the configured logger.

Parameters
----------
log_line : dict[str, Any]
Dictionary containing log record details with keys:
- 'level': Logging level
- 'filename': Source filename
- 'line': Line number
- 'msg': Log message
- 'function': Source function name
- 'extra': Additional context
- 'timestamp': Original log creation time

Notes
-----
Bypasses standard logging flow by directly creating and handling a log record.
Preserves original timestamp and source information.
"""
record = self._logger.makeRecord(
name=self.name,
level=log_line["level"],
fn=log_line["filename"],
lno=log_line["line"],
msg=log_line["msg"],
args=(),
exc_info=None,
func=log_line["function"],
extra=log_line["extra"],
)
record.created = log_line["timestamp"]
self._logger.handle(record)

def _add_log_record_to_buffer(
self,
level: int,
msg: object,
args: object,
exc_info: logging._ExcInfoType = None,
stack_info: bool = False,
extra: Mapping[str, object] | None = None,
) -> None:
"""
Add log record to buffer with intelligent tracer ID handling.

Parameters
----------
level : int
Logging level of the record.
msg : object
Log message to be recorded.
args : object
Additional arguments for the log message.
exc_info : logging._ExcInfoType, optional
Exception information for the log record.
stack_info : bool, optional
Whether to include stack information.
extra : Mapping[str, object], optional
Additional contextual information for the log record.

Raises
------
InvalidBufferItem
If the log record cannot be added to the buffer.

Notes
-----
Handles special first invocation buffering and migration of log records
between different tracer contexts.
"""
# Determine tracer ID, defaulting to first invoke marker
tracer_id = get_tracer_id()

if tracer_id:
log_record: dict[str, Any] = _create_buffer_record(level=level, msg=msg, args=args, extra=extra)
self._buffer_cache.add(tracer_id, log_record)

def flush_buffer(self) -> None:
"""
Flush all buffered log records associated with current execution.

Notes
-----
Retrieves log records for current trace from buffer
Immediately processes and logs each record
Warning if some cache was evicted in that execution
Clears buffer after complete processing

Raises
------
Any exceptions from underlying logging or buffer mechanisms
will be propagated to caller
"""
tracer_id = get_tracer_id()

# Flushing log without a tracer id? Return
if not tracer_id:
return

# is buffer empty? return
buffer = self._buffer_cache.get(tracer_id)
if not buffer:
return

# Process log records
for log_line in buffer:
self._create_and_flush_log_record(log_line)

# Has items evicted?
if self._buffer_cache.has_items_evicted(tracer_id):
warnings.warn(
message="Some logs are not displayed because they were evicted from the buffer. "
"Increase buffer size to store more logs in the buffer",
category=PowertoolsUserWarning,
stacklevel=2,
)

# Clear the entire cache
self._buffer_cache.clear()


def set_package_logger(
level: str | int = logging.DEBUG,
Expand Down
Loading
Loading