Skip to content

perf(grpo): use JSONL append-only format for training log #47

@abrichr

Description

@abrichr

Problem

trainer.py:674-677 rewrites the entire JSON training log on every step. With 1000 steps, the file grows continuously and the O(N) full rewrite becomes slow.

Proposed Fix

Switch to JSONL (one JSON line per entry, append-only) which is O(1) per write and easier to process with streaming tools.

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance improvement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions