Skip to content

bug: MemoryEntry display crashes with UnicodeEncodeError on Windows cp125x consoles #321

Description

@farizanjum

Summary

MemoryEntry.__str__() embeds literal Unicode box-drawing characters (U+2514 └, U+2500 ─, U+251C ├) in its output. When display=True (the default), Console().print(panel) writes these to sys.stdout. On Windows installations where sys.stdout.encoding is a cp125x variant, these codepoints have no encoding mapping and raise UnicodeEncodeError, crashing the simulation.

This affects all memory backends (STMemory, STLTMemory, LTMemory) since they all call MemoryEntry.display().

Reproduction

# repro.py — no API keys needed, no LLM calls, pure rendering
from rich.console import Console
from rich.panel import Panel

# Simulate a cp1252 stdout (default on Windows en-US when piped/redirected)
import io, sys
buf = io.BytesIO()
sys.stdout = io.TextIOWrapper(buf, encoding="cp1252", errors="strict")

# Replicate MemoryEntry.__str__() output
content = "[bold cyan][Observation][/bold cyan]\n   [blue]\u2514\u2500[/blue] [cyan]price :[/cyan] $20"

panel = Panel(content, title="Step 1 | Agent 0", title_align="left",
              border_style="bright_blue", padding=(0, 1))
Console().print(panel)
stdout encoding: cp1252
+- Step 1 | Test 0 ------...---+     <- Rich degrades ITS borders to ASCII (safe_box)
| [Observation]               |
|    CRASHES: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

Rich's safe_box correctly degrades its own panel borders to ASCII. But the box-drawing characters in __str__() are literal content text, not Rich decoration -- Rich cannot degrade them, so they pass through to sys.stdout.write() and fail.

Expected behavior

Memory display renders without crashing on all platforms and all Windows code pages.

Actual behavior

UnicodeEncodeError: 'charmap' codec can't encode character '\u2514' -- simulation crashes.

Who is affected

sys.stdout.encoding on Windows depends on the system locale. The box-drawing characters used (└─├) crash on all cp125x variants:

Encoding Region └─├ (U+2514/2500/251C)
cp1252 Western Europe (en, fr, de, es, pt, it, nl, ...) CRASH
cp1250 Central Europe (cs, pl, hu, sk, ro, hr) CRASH
cp1251 Cyrillic (ru, bg, uk) CRASH
cp1253 Greek CRASH
cp1254 Turkish CRASH
cp1255 Hebrew CRASH
cp1256 Arabic CRASH
cp874 Thai CRASH
cp437 / cp850 DOS/legacy US-EU console safe
cp932 / cp936 / cp949 Japanese / Chinese / Korean safe
cp65001 UTF-8 (opt-in) safe

This triggers in common scenarios: piping output, running inside VS Code / PyCharm / Jupyter, CI runners -- anywhere Python falls back to locale.getpreferredencoding() which returns the cp125x variant.

Options considered

1. Replace with • (U+2022 bullet) -- rejected

• is safe in cp125x but crashes on cp437, cp850, cp932, cp936, cp949 (DOS-legacy and Asian Windows). This trades one crash for another.

Encoding └─├ • (U+2022)
cp1252 (Western Windows) CRASH safe
cp437 (DOS/legacy US) safe CRASH
cp932 (Japanese) safe CRASH
cp936 (Chinese) safe CRASH

2. Reconfigure sys.stdout to UTF-8 -- rejected

Libraries should not mutate global sys.stdout encoding. Side effects on user code, not guaranteed to work (buffer may not support reconfigure), and does not fix the root cause.

3. Wrap console.print() in try/except -- rejected

Hides the root cause, swallows the error silently, adds complexity. The user sees no output instead of a crash -- still broken, just quieter.

4. Replace with ASCII tree markers (+--, |--) -- recommended

+, -, | are pure ASCII (bytes 0x2B, 0x2D, 0x7C). Safe in every encoding ever created -- cp125x, cp437, cp9xx, cp65001, all of them. Preserves the tree structure that └─/├─ provided. This is the same rendering that tree --charset=ASCII and npm list use.

Before:

[Observation]
   └─ price : $20
   └─ seller : Bob
[Action]
   ├─ (1)
   │  └─ offer : $25
   ├─ (2)

After:

[Observation]
   +-- price : $20
   +-- seller : Bob
[Action]
   |-- (1)
   |   +-- offer : $25
   |-- (2)

7 lines changed in memory.py, zero logic change, zero behavior change.

Proposed fix

File: mesa_llm/memory/memory.py, MemoryEntry.__str__() (lines 52-90)

Replace └─ with +-- and ├─ with |--. Add a test verifying str(entry) output is encodable across all major Windows code pages.

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions