Add MaxDepth property to CborReader and CborWriter

[Copilot]

Adds a configurable nesting-depth limit to System.Formats.Cbor, guarding the reader against excessive work/stack pressure from deeply nested untrusted input and the writer against runaway/cyclic encoders. Introduces CborReaderOptions/CborWriterOptions (per the approved API in issue #128087) rather than stacking another [EditorBrowsable(Never)] ctor overload.

New public API

namespace System.Formats.Cbor;

public partial class CborReader
{
    public CborReader(ReadOnlyMemory<byte> data, CborReaderOptions? options);
    public int MaxDepth { get; }
}

public sealed class CborReaderOptions
{
    public bool AllowMultipleRootLevelValues { get; set; }
    public CborConformanceMode ConformanceMode { get; set; }
    public int MaxDepth { get; set; }
}

public partial class CborWriter
{
    public CborWriter(CborWriterOptions? options);
    public int MaxDepth { get; }
}

public sealed class CborWriterOptions
{
    public bool AllowMultipleRootLevelValues { get; set; }
    public CborConformanceMode ConformanceMode { get; set; }
    public bool ConvertIndefiniteLengthEncodings { get; set; }
    public int InitialCapacity { get; set; }
    public int MaxDepth { get; set; }
}

Changes

• Options classes — sealed CborReaderOptions/CborWriterOptions with eager setter validation (ConformanceMode and negative MaxDepth/InitialCapacity throw ArgumentOutOfRangeException). Default ConformanceMode is Strict via backing field initializer.
• Constructors — new nullable-options ctors; legacy ctors chain through private ctors. null options yields legacy defaults.
• Depth semantics — default MaxDepth is 64; options.MaxDepth == 0 means "use library default". The MaxDepth property reports the effective limit, not the raw option value.
• Enforcement — EnsureMaxDepthNotExceeded() is invoked after the non-consuming PeekInitialByte but before AdvanceBuffer/PushDataItem, so a rejected frame doesn't move BytesConsumed and the reader locks onto the error state correctly (supports checkpoint rollback). Applies to ReadStartArray/ReadStartMap/indefinite-length string starts, and transitively to SkipValue/ReadEncodedValue; the writer mirrors this on the WriteStart* paths. Tags do not push a frame and do not count toward depth.
• Exception types — reader throws CborContentException (depth is a property of untrusted content); writer throws InvalidOperationException (programming error, matching Utf8JsonWriter).
• Writer doc disclaimer — CborWriterOptions.MaxDepth documents the narrow intent (detect runaway/cyclic encoders) and explicitly disclaims being a defense against already-hydrated hostile object graphs.
• Updated ref surface, resource strings, and csproj; added reader/writer tests. The existing 50,000-deep SkipValue stack-overflow test now opts into a matching MaxDepth.

Usage

// Reader: cap depth on untrusted input.
var reader = new CborReader(data, new CborReaderOptions { MaxDepth = 32 });
// the 33rd nested ReadStartArray/Map throws CborContentException.

// Writer: detect runaway encoders / cycles.
var writer = new CborWriter(new CborWriterOptions { MaxDepth = 1 });
writer.WriteStartArray(1);
writer.WriteStartArray(1); // throws InvalidOperationException

Open question

DefaultMaxDepth is set to 64 (matching System.Text.Json and the reference branch). An earlier proposal draft mentioned 1000 — worth a reviewer confirmation.

[eiriktsarpalis]

@bartonjs should we flag this as a breaking change?

[Copilot]

Pull request overview

This PR extends System.Formats.Cbor with a configurable maximum nesting depth for both CborReader and CborWriter, exposed via new nullable-options constructors plus MaxDepth properties. The change adds enforcement points on container-opening operations (arrays, maps, and indefinite-length strings) and introduces new CborReaderOptions / CborWriterOptions public types.

Changes:

• Added new public options types (CborReaderOptions, CborWriterOptions) and new nullable-options constructors, with MaxDepth exposed on reader/writer instances.
• Implemented MaxDepth enforcement in reader/writer container entry points via EnsureMaxDepthNotExceeded(), with reader throwing CborContentException and writer throwing InvalidOperationException.
• Added resource strings, updated ref surface + csproj compile items, and added/updated tests to validate defaulting and enforcement behavior.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.cs	Adds DefaultMaxDepth, MaxDepth property, nullable-options ctor, and depth enforcement helper.
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.Array.cs	Enforces MaxDepth on ReadStartArray().
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.Map.cs	Enforces MaxDepth on ReadStartMap().
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.String.cs	Enforces MaxDepth on indefinite-length string start APIs.
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReaderOptions.cs	New public options class for reader configuration, including MaxDepth.
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Writer/CborWriter.cs	Adds DefaultMaxDepth, MaxDepth property, nullable-options ctor, and depth enforcement helper.
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Writer/CborWriter.Array.cs	Enforces MaxDepth on WriteStartArray*.
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Writer/CborWriter.Map.cs	Enforces MaxDepth on WriteStartMap*.
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Writer/CborWriter.String.cs	Enforces MaxDepth on indefinite-length string start APIs.
src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Writer/CborWriterOptions.cs	New public options class for writer configuration, including MaxDepth and InitialCapacity.
src/libraries/System.Formats.Cbor/src/Resources/Strings.resx	Adds reader/writer MaximumDepthExceeded resource strings.
src/libraries/System.Formats.Cbor/src/System.Formats.Cbor.csproj	Includes new options source files in compilation list.
src/libraries/System.Formats.Cbor/ref/System.Formats.Cbor.cs	Updates reference surface with new public APIs/types.
src/libraries/System.Formats.Cbor/tests/Reader/CborReaderTests.cs	Adds tests for reader MaxDepth reporting/defaulting/enforcement.
src/libraries/System.Formats.Cbor/tests/Reader/CborReaderTests.SkipValue.cs	Updates deep-nesting SkipValue test to opt into a matching MaxDepth.
src/libraries/System.Formats.Cbor/tests/Writer/CborWriterTests.cs	Adds tests for writer MaxDepth reporting/defaulting/enforcement and options flow-through.

[bartonjs]

This PR is unwelcome, as I've already written it.