fix: replace UnicodeDecodeError to prevent large payload logging

[CyMule]

Replace UnicodeDecodeError with UnprocessableEntityError in encoding detection to avoid logging entire file contents. UnicodeDecodeError.object automatically stores complete input data, causing memory issues with large files in logging and error reporting systems.

[yuming-long]

why from None

[CyMule]

APM would end up capturing the original exception which would contain the full bytes we tried to decode. The None prevents the exception chaining and importantly prevents the bytes from being passed to APM

[yuming-long]

gotcha thanks!

[yuming-long]

@claude review this?

[claude]

Claude finished @yuming-long's task —— View job

---

Code Review In Progress

Todo List:

• Read and analyze all changed files
• Review the encoding.py changes for correctness
• Address the "from None" review comment question
• Check test coverage and quality
• Verify the memory issue fix is properly implemented
• Provide comprehensive review feedback

Review Findings

✅ Excellent Fix for Memory Issue

The core change replacing UnicodeDecodeError with UnprocessableEntityError is well-justified. The PR correctly identifies that UnicodeDecodeError.object automatically stores the complete input data, which can cause serious memory issues with large files in logging systems.

✅ "from None" Usage is Appropriate

Addressing @yuming-long's question on line 111:

The from None syntax is correct here because:

1. Prevents exception chaining - Avoids keeping references to the original UnicodeDecodeError which contains the large file data in its .object attribute
2. Memory safety - This is the core purpose of the PR - to prevent memory cascade issues
3. Clean error messages - Users get clear, actionable error messages without confusing exception chains

✅ Comprehensive Test Coverage

The new tests in test_encoding.py are exceptional:

• Tests both failure scenarios (charset detection + decode failure)
• Memory safety verification - Explicitly checks exceptions don't have .object attribute
• Size assertions - Verifies exceptions have small memory footprint (<10KB)
• Realistic test data - Uses 1MB of invalid data to simulate real-world conditions

✅ Code Quality

• Clear, detailed comments explaining the rationale
• Consistent error messaging
• Proper changelog documentation
• Appropriate version bump

Minor Suggestions

1. Consider adding a brief docstring example in the functions showing the new exception type
2. The test could potentially verify that the original UnicodeDecodeError would have been larger (for demonstration)

Overall Assessment

LGTM ✅ This is a well-crafted fix that properly addresses a legitimate memory issue with thorough testing and clear documentation.

---

[mateuszkuprowski]

LGTM!
Love what you've done here!

[yuming-long]

🎉