Increase history length of flight recorder

[mhofman]

What is the Problem Being Solved?

The flight recorder is very useful to gather diagnostics from validators experiencing certain type of issues (e.g. AppHash failures), however because it's a ring buffer of limited size, we can quickly lose useful context. In particular during a restart, vats are replayed, which generate a lot of slog output that completely fills up the ring buffer.

During the debugging of #8423, we also noticed that the slog extracted from the flight recorder was much smaller than the capacity.

Description of the Design

A simple change would be to increase the circularBufferSize option. While this is technically a configurable option, no-one actually sets it in code, and it doesn't use an environment variable for configuration like other slog sender config. The current default value is 100 MB.

Security Considerations

None

Scaling Considerations

None

Test Plan

update the frcat tool and manually test

Upgrade Considerations

None

[mhofman]

This would have potentially helped diagnose #8423 further if we could have read the previous run's slogs.

[ivanlei]

Could potentially clobber flight recorder on startup and avoid complexity of increasing flight recorder size.