[API Proposal]: Add capability in `PEReader` to reject COFF files.

[teo-tsirpanis]

Background and motivation

When PEReader reads an image that does not have a PE header, it tries to read it as a COFF file. However, because COFF files do not have a signature, the reader will not immediately fail when reading files that are not COFF files, which has misleading behavior by not failing, and wastes memory to store section headers. This has affected both first-party (#112653 (comment)) and third-party users (#48419).

I am proposing to add an option to cause PEReader to immediately throw if the image does not contain a PE header and not attempt to further read it.

API Proposal

namespace System.Reflection.PortableExecutable;

public enum PEStreamOptions
{
    RequirePEHeader = 16,
}

API Usage

using var pe = new PEReader(File.OpenRead("MyFile.dll"), PEStreamOptions.RequirePEHeader);

Alternative Designs

• The name of the flag could be different.
• Should we also add more overloads to cover reading images from contiguous memory?
• Should we also add a PEHeaders(Stream peStream, int size, bool isLoadedImage, bool requirePEHeader) constructor overload?

Risks

The API might be too niche or too hard to discover. Users can do the PE signature validation themselves before creating the PEReader; it's quite simple to implement.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata
See info in area-owners.md if you want to be subscribed.

[MichalStrehovsky]

It's also possible to do a bit more validation that this file makes sense as a COFF object:

• SizeOfOptionalHeader needs to be zero in a COFF file (this would be enough to fix the repro in PEReader does not throw BadImageFormatException for some invalid PE files #48419)
• PointerToSymbolTable must be less than the file size
• Characteristics field has several bits that must be zero and a couple other bits that must be zero in object files. It effectively always is all-zero, but that's not guaranteed.

One could also go further and add a range check for the NumberOfSections and then validate each section has a zero VirtualAddress (must be zero in object file).

It's not as good as looking for magic numbers, but better than nothing.

[steveharter]

It's also possible to do a bit more validation that this file makes sense as a COFF object:

Yes thanks. We should start by adding validation to reject a non-COFF (after knowing the file is not a PE) then add the enum if we still need it.