Redesign and improve rendering through c_render with ffmpeg/xvfb

[daphne-cornelisse]

Problem

Headless rendering is an important feature since many GPUs - such as those on training clusters - have no display. While headless support already existed in visualize.c via ffmpeg, it depended on drivenet.h, a C reimplementation of the torch network. This created a maintenance burden and a source of bugs, particularly when experimenting with different architectures. The rendering code had also grown bloated and difficult to follow.

Proposed solution

This PR removes the dependency on drivenet.c entirely, moving inference to the Python-side torch policy instead. Rendering is now driven from Python, with c_render called directly in the eval loop alongside vecenv.step.

The rendering code is also simplified. The key observation is that the Client struct can be initialized in two modes: a standard pop-up window mode, and a headless mode that forks an ffmpeg process and pipes raw RGBA frames to it for encoding. Window dimensions in headless mode are derived from the actual map bounds, and the orthographic camera is centered on the map's bounding box. The video is finalized cleanly when vecenv.close() closes the client. There is no difference in SPS compared to the previous implementation.

Usage

Render with pre-trained cpt / just once

• 1. Set the render_mode in .ini

; Render mode options:
; 0:"window" = pop-up raylib window (original)
; 1:"headless" = off-screen; frames piped to ffmpeg (recommended for training)
render_mode = 1

• 2. Run eval

puffer eval puffer_drive

This essentially just runs env.render() a couple of times, then closes it.

View mode

The rendering view can be configured through the view mode in Python (snapshot from drive.py):

class RenderView(IntEnum):
    FULL_SIM_STATE = 0  # Orthographic top-down, fully observable simulator state
    BEV_AGENT_OBS = 1  # Orthographic top-down, only show what the selected agent can observe
    AGENT_PERSP = 2  # Third-person perspective following selected agent

[...]

def render(
        self, 
        view_mode: RenderView = RenderView.FULL_SIM_STATE, 
        draw_traces: bool = True,
        env_id: int = 0
    ):
        binding.vec_render(self.c_envs, int(view_mode), draw_traces, env_id)

During training

Check out the settings under [eval] inside drive.ini

Summary of improvements

• [Major] Remove dependence on drivenet.h.
• [Major] Rendering does not rely on the stored .bin file anymore but uses the current uncompiled torch policy directly.
• [Major] Refactor HumanReplayEvaluator into a more general-purpose Evaluator class with visual eval (rendering) supported. This class can be easily extended to support other functionalities, such as rendering scenarios with high collision scores.
• [Major] Simplify rendering code.
• [Minor] Better color scheme and agent visibility.
• Deletes old visualize.c file (no longer used since all logic is self-contained in c_render())

Sharp edges

---

‼️ Raylib is not thread safe - make sure to complete the entire rollout before opening a new one, i.e., do env_1.close() before starting env_2.render() -> otherwise it will fail.

---

New look

• Collect stats from desired eval settings and render randomly selected scenarios (can be changed later to render particular scenes)

• Controlled agents are colored by type for visibility
• Expert replays (fixed) are colored in pastel green

1dd47642-c547-4e2a-b96c-902d074f31a3.mp4

59e55c55-25f0-427d-8616-5a72d8ad1d9f.mp4

[greptile-apps]

_{10 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}