Skip to content

feat(logger): add logger buffer feature #6060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Mar 7, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
d1a7a58
First commit - Adding Buffer and Config classes
leandrodamascena Feb 9, 2025
6c4ef9c
First commit - Adding Buffer and Config classes
leandrodamascena Feb 9, 2025
e2dd0c4
First commit - Adding Buffer and Config classes
leandrodamascena Feb 9, 2025
511f147
First commit - Adding nox tests
leandrodamascena Feb 9, 2025
ebf49db
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 9, 2025
d24a258
Adding new parameter to the constructor
leandrodamascena Feb 9, 2025
0893e14
Starting tests
leandrodamascena Feb 10, 2025
505eb98
Adding initial logic + test
leandrodamascena Feb 13, 2025
dbfd708
Adding initial logic + test
leandrodamascena Feb 13, 2025
8a4e7bf
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 13, 2025
93f4a13
Adding initial logic + test
leandrodamascena Feb 13, 2025
35acff5
Unit tests for log comparasion
leandrodamascena Feb 13, 2025
ed3902d
Fixing logic to handle log flush
leandrodamascena Feb 13, 2025
e39229a
Start flushing logs
leandrodamascena Feb 13, 2025
27b1ce9
Adding more logic
leandrodamascena Feb 13, 2025
7b06ac9
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 17, 2025
69eda82
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 17, 2025
10d81d9
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 19, 2025
a98ee0a
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 20, 2025
58e26ae
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 21, 2025
e97d065
Adding more logic
leandrodamascena Feb 21, 2025
8b30fe5
Adding more logic
leandrodamascena Feb 21, 2025
5406a77
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 24, 2025
1893216
Refactoring buffer class
leandrodamascena Feb 24, 2025
ce3804e
Refactoring buffer class
leandrodamascena Feb 24, 2025
7f49bb4
Refactoring buffer class
leandrodamascena Feb 25, 2025
351d856
More refactor
leandrodamascena Feb 28, 2025
f7eeb2e
Merge branch 'develop' into feat/buffer-log
leandrodamascena Feb 28, 2025
7d98138
Adding more coverage + documentation
leandrodamascena Feb 28, 2025
0c8cb3b
Adding decorator
leandrodamascena Feb 28, 2025
c977904
Refactoring variables name
leandrodamascena Mar 1, 2025
c9adbee
Adding more tests + propagating buffer configuration
leandrodamascena Mar 1, 2025
699a3d8
Adding exception fields
leandrodamascena Mar 2, 2025
49f308b
Adding e2e tests
leandrodamascena Mar 2, 2025
fa5c7f1
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 2, 2025
b1d81b5
sys.getframe is more performatic
leandrodamascena Mar 2, 2025
feaea2b
Chaging the max size
leandrodamascena Mar 3, 2025
a46e2ff
Removing unecessary files
leandrodamascena Mar 3, 2025
9368459
Fix tests
leandrodamascena Mar 3, 2025
1167850
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 5, 2025
f9db4cd
Flushing logs when log line is bigger than buffer size
leandrodamascena Mar 5, 2025
f30eed0
Flushing logs when log line is bigger than buffer size
leandrodamascena Mar 5, 2025
452cf91
Adding documentation
leandrodamascena Mar 5, 2025
2d8eb39
Adding documentation
leandrodamascena Mar 5, 2025
46a9ab9
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 5, 2025
a5a12a5
Adding documentation
leandrodamascena Mar 6, 2025
46fc1df
Addressing Andrea's feedback
leandrodamascena Mar 6, 2025
b1e2f99
Addressing Andrea's feedback
leandrodamascena Mar 6, 2025
4717cdc
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 6, 2025
f5618ea
Addressing Andrea's feedback
leandrodamascena Mar 6, 2025
1da9bf2
Refactoring parameters name + child logger + tests
leandrodamascena Mar 6, 2025
a64df93
Improve documentation
leandrodamascena Mar 6, 2025
078b33c
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 6, 2025
bd7084f
Merge branch 'develop' into feat/buffer-log
leandrodamascena Mar 7, 2025
5c9eb79
Improve docstring
leandrodamascena Mar 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Adding documentation
  • Loading branch information
leandrodamascena committed Mar 5, 2025
commit 2d8eb3973909ee8b516cb03b2117e87fe98d3fcc
9 changes: 0 additions & 9 deletions aws_lambda_powertools/logging/buffer/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ def _validate_inputs(
max_size: int,
minimum_log_level: str,
flush_on_error: bool,
compress: bool,
) -> None:
"""
Validate configuration inputs.
Expand All @@ -65,9 +64,6 @@ def _validate_inputs(
if not isinstance(flush_on_error, bool):
raise ValueError("flush_on_error must be a boolean")

if not isinstance(compress, bool):
raise ValueError("compress must be a boolean")

@property
def max_size(self) -> int:
"""Maximum buffer size in bytes."""
Expand All @@ -82,8 +78,3 @@ def minimum_log_level(self) -> str:
def flush_on_error(self) -> bool:
"""Flag to flush buffer on error."""
return self._flush_on_error

@property
def compress(self) -> bool:
"""Flag to compress buffered logs."""
return self._compress
28 changes: 9 additions & 19 deletions docs/core/logger.md
Original file line number Diff line number Diff line change
Expand Up @@ -562,88 +562,78 @@ Use the `@logger.inject_lambda_context` decorator to automatically flush buffere

##### Flushing manually

<center>
```mermaid
sequenceDiagram
participant Client
participant Lambda
participant Logger
participant CloudWatch

Client->>Lambda: Invoke Lambda
Lambda->>Logger: Initialize with DEBUG level buffering
Logger-->>Lambda: Logger buffer ready

Lambda->>Logger: logger.debug("First debug log")
Logger-->>Logger: Buffer first debug log

Lambda->>Logger: logger.info("Info log")
Logger->>CloudWatch: Directly log info message

Lambda->>Logger: logger.debug("Second debug log")
Logger-->>Logger: Buffer second debug log

Lambda->>Logger: logger.flush_buffer()
Logger->>CloudWatch: Flush buffered debug logs to CloudWatch

Lambda->>Client: Return execution result

```
<i>Flushing buffer manually</i>
</center>

##### Flushing on error

<center>
```mermaid
sequenceDiagram
participant Client
participant Lambda
participant Logger
participant CloudWatch

Client->>Lambda: Invoke Lambda
Lambda->>Logger: Initialize with DEBUG level buffering
Logger-->>Lambda: Logger buffer ready

Lambda->>Logger: logger.debug("First log")
Logger-->>Logger: Buffer first debug log

Lambda->>Logger: logger.debug("Second log")
Logger-->>Logger: Buffer second debug log

Lambda->>Logger: logger.debug("Third log")
Logger-->>Logger: Buffer third debug log

Lambda->>Lambda: Exception occurs
Lambda->>Logger: logger.error("Error details")
Logger->>CloudWatch: Send buffered debug logs
Logger->>CloudWatch: Send error log

Lambda->>Client: Raise exception
```
<i>Flushing buffer when an error happens</i>
</center>

##### Flushing when uncaught exception happens

<center>
```mermaid
sequenceDiagram
participant Client
participant Lambda
participant Logger
participant CloudWatch

Client->>Lambda: Invoke Lambda
Lambda->>Logger: Using decorator @logger.inject_lambda_context(flush_buffer_on_uncaught_error=True)
Logger-->>Lambda: Logger context injected

Lambda->>Logger: logger.debug("First log")
Logger-->>Logger: Buffer first debug log

Lambda->>Logger: logger.debug("Second log")
Logger-->>Logger: Buffer second debug log

Lambda->>Lambda: Uncaught Exception
Lambda->>CloudWatch: Automatically send buffered debug logs

Lambda->>Client: Raise uncaught exception

```
<i>Flushing buffer when an uncaught exception happens</i>
</center>

#### Common buffering questions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ def test_default_configuration():
assert config_buffer.max_size == 20480
assert config_buffer.minimum_log_level == "DEBUG"
assert config_buffer.flush_on_error is True
assert config_buffer.compress is False


def test_custom_configuration():
Expand All @@ -20,14 +19,12 @@ def test_custom_configuration():
max_size=51200,
minimum_log_level="WARNING",
flush_on_error=False,
compress=True,
)

# THEN configuration is set with provided values
assert config_buffer.max_size == 51200
assert config_buffer.minimum_log_level == "WARNING"
assert config_buffer.flush_on_error is False
assert config_buffer.compress is True


def test_invalid_max_size_negative():
Expand Down Expand Up @@ -79,13 +76,3 @@ def test_invalid_flush_on_error():
with pytest.raises(ValueError, match="flush_on_error must be a boolean"):
# THEN a ValueError is raised
LoggerBufferConfig(flush_on_error=invalid_flush_on_error)


def test_invalid_compress():
# GIVEN an invalid compress type
invalid_compress = "False"

# WHEN creating a LoggerBufferConfig / THEN
with pytest.raises(ValueError, match="compress must be a boolean"):
# THEN a ValueError is raised
LoggerBufferConfig(compress=invalid_compress)
Loading