Windows SDK tools do not support working with heavy portable symbols files (Part 2)

[LunarWhisper]

Description

In a large project containing a massive number of types (~14 000), the generated .pdb file becomes incompatible with tools like WinDBG and symstore.exe once it reaches a certain size. However, the IDE handles these files without any issues.

This is a continuation of the issue in which we found a similar problem with the number of files greater than UInt16.MaxValue, but now it fires on a smaller number of files.
#108833

Reproduction Steps

We don't know how or why this happens.

Expected behavior

SYMSTORE MESSAGE: Copying Core.pdb to ...
SYMSTORE: Number of files stored = 1
SYMSTORE: Number of errors = 0
SYMSTORE: Number of files ignored = 0

.pdb saved on target

Actual behavior

On small projects everything works as usual without any errors.

On large real project:

SYMSTORE: Number of files stored = 0
SYMSTORE: Number of errors = 0
SYMSTORE: Number of files ignored = 1

AccessViolation thrown inside:

0:000> gn
ModLoad: 00007ff9`1aad0000 00007ff9`1ab4b000   C:\WINDOWS\System32\bcryptPrimitives.dll
(96c.542c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
dbghelp!DBI1::addFileInfo+0x125:
00007ff8`632435d5 42893ca0        mov     dword ptr [rax+r12*4],edi ds:0000023e`21c39000=????????
0:004> k
 # Child-SP          RetAddr               Call Site
00 00000049`7a07ba50 00007ff8`6325d3fd     dbghelp!DBI1::addFileInfo+0x125
01 00000049`7a07bac0 00007ff8`632597ff     dbghelp!Mod1::processC13+0x7ad
02 00000049`7a07bd70 00007ff8`6325ad7b     dbghelp!Mod1::fProcessSyms+0x11ff
03 00000049`7a07be30 00007ff8`632504e7     dbghelp!Mod1::fUpdateSyms+0x1b
04 00000049`7a07be60 00007ff8`6323638f     dbghelp!Mod1::Close+0x207
05 00000049`7a07bee0 00007ff8`63221dd4     dbghelp!PortablePDB::ConvertPortablePDB+0x3df
06 00000049`7a07c820 00007ff8`632211cd     dbghelp!PDB1::OpenPortablePDB+0x84
07 00000049`7a07c860 00007ff8`63221585     dbghelp!PDB1::OpenEx2W+0x5ed
08 00000049`7a07c920 00007ff8`6323273b     dbghelp!PDB::OpenEx2W+0x35
09 00000049`7a07c970 00007ff8`632898c7     dbghelp!PDBCommon::Open2W+0x2b
0a 00000049`7a07c9c0 00007ff8`63289729     dbghelp!CDiaDataSource::loadDataFromPdbHelper+0x57
0b 00000049`7a07ca10 00007ff8`6332a2fd     dbghelp!CDiaDataSource::loadDataFromPdb+0x19
0c 00000049`7a07ca50 00007ff8`63336e9a     dbghelp!DiaOpenPdbEx+0x201
0d 00000049`7a07cb00 00007ff8`633363d9     dbghelp!diaOpenPdb+0x56
0e 00000049`7a07cb60 00007ff8`6333336b     dbghelp!diaLocatePdb+0x17d
0f 00000049`7a07d8c0 00007ff8`6335837a     dbghelp!diaGetPdb+0x29f
10 00000049`7a07db90 00007ff8`6335cce8     dbghelp!GetDebugData+0x2de
11 00000049`7a07de20 00007ff8`6335c99b     dbghelp!modloadWorker+0x2e0
12 00000049`7a07e320 00007ff8`63358ff4     dbghelp!modload+0xff
13 00000049`7a07e5c0 00007ff8`63352b30     dbghelp!LoadModule+0x608
14 00000049`7a07eae0 00007ff6`fc5f58cf     dbghelp!SymLoadModuleEx+0x80
15 00000049`7a07eb50 00007ff6`fc5f025f     SymStore!SymCommonGetFileInformation+0x13f
16 00000049`7a07f290 00007ff9`1b09257d     SymStore!Add1ChildThread+0x57f
17 00000049`7a07fed0 00007ff9`1cf6af28     KERNEL32!BaseThreadInitThunk+0x1d
18 00000049`7a07ff00 00000000`00000000     ntdll!RtlUserThreadStart+0x28

0:004> gn
(96c.542c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
ntdll!RtlpLocateRelatedBlocks+0xa7:
00007ff9`1d02743f 488b7a10        mov     rdi,qword ptr [rdx+10h] ds:003d06dd`003d067b=????????????????
0:004> k
 # Child-SP          RetAddr               Call Site
00 00000049`7a079be0 00007ff9`1d025dde     ntdll!RtlpLocateRelatedBlocks+0xa7
01 00000049`7a079c00 00007ff9`1d031de5     ntdll!RtlpHpHeapHandleError+0x6e
02 00000049`7a079c30 00007ff9`1d025a93     ntdll!RtlpLogHeapFailure+0x45
03 00000049`7a079c60 00007ff9`1cf50793     ntdll!RtlpAnalyzeHeapFailure+0x30f
04 00000049`7a079cc0 00007ff9`1cf4d239     ntdll!RtlpAllocateHeap+0x1663
05 00000049`7a079f20 00007ff9`1d014772     ntdll!RtlpAllocateHeapInternal+0x6c9
06 00000049`7a07a020 00007ff9`1cfce0da     ntdll!RtlDebugAllocateHeap+0x102
07 00000049`7a07a0c0 00007ff9`1cf4d239     ntdll!RtlpAllocateHeap+0x7efaa
08 00000049`7a07a320 00007ff9`1a5e33ae     ntdll!RtlpAllocateHeapInternal+0x6c9
09 00000049`7a07a420 00007ff9`1a6220cb     ucrtbase!_calloc_base+0x4e
0a 00000049`7a07a450 00007ff9`1a622051     ucrtbase!_vcrt_getptd_noexit+0x63
0b 00000049`7a07a480 00007ff9`1a6148da     ucrtbase!_vcrt_getptd+0x9
0c 00000049`7a07a4b0 00007ff9`1cfb51df     ucrtbase!_CxxFrameHandler3+0x2a
0d 00000049`7a07a500 00007ff9`1cf2e866     ntdll!RtlpExecuteHandlerForException+0xf
0e 00000049`7a07a530 00007ff9`1cfb41ce     ntdll!RtlDispatchException+0x286
0f 00000049`7a07ac80 00007ff8`632435d5     ntdll!KiUserExceptionDispatch+0x2e
10 00000049`7a07ba50 00007ff8`6325d3fd     dbghelp!DBI1::addFileInfo+0x125
11 00000049`7a07bac0 00007ff8`632597ff     dbghelp!Mod1::processC13+0x7ad
12 00000049`7a07bd70 00007ff8`6325ad7b     dbghelp!Mod1::fProcessSyms+0x11ff
13 00000049`7a07be30 00007ff8`632504e7     dbghelp!Mod1::fUpdateSyms+0x1b
14 00000049`7a07be60 00007ff8`6323638f     dbghelp!Mod1::Close+0x207
15 00000049`7a07bee0 00007ff8`63221dd4     dbghelp!PortablePDB::ConvertPortablePDB+0x3df
16 00000049`7a07c820 00007ff8`632211cd     dbghelp!PDB1::OpenPortablePDB+0x84
17 00000049`7a07c860 00007ff8`63221585     dbghelp!PDB1::OpenEx2W+0x5ed
18 00000049`7a07c920 00007ff8`6323273b     dbghelp!PDB::OpenEx2W+0x35
19 00000049`7a07c970 00007ff8`632898c7     dbghelp!PDBCommon::Open2W+0x2b
1a 00000049`7a07c9c0 00007ff8`63289729     dbghelp!CDiaDataSource::loadDataFromPdbHelper+0x57
1b 00000049`7a07ca10 00007ff8`6332a2fd     dbghelp!CDiaDataSource::loadDataFromPdb+0x19
1c 00000049`7a07ca50 00007ff8`63336e9a     dbghelp!DiaOpenPdbEx+0x201
1d 00000049`7a07cb00 00007ff8`633363d9     dbghelp!diaOpenPdb+0x56
1e 00000049`7a07cb60 00007ff8`6333336b     dbghelp!diaLocatePdb+0x17d
1f 00000049`7a07d8c0 00007ff8`6335837a     dbghelp!diaGetPdb+0x29f
20 00000049`7a07db90 00007ff8`6335cce8     dbghelp!GetDebugData+0x2de
21 00000049`7a07de20 00007ff8`6335c99b     dbghelp!modloadWorker+0x2e0
22 00000049`7a07e320 00007ff8`63358ff4     dbghelp!modload+0xff
23 00000049`7a07e5c0 00007ff8`63352b30     dbghelp!LoadModule+0x608
24 00000049`7a07eae0 00007ff6`fc5f58cf     dbghelp!SymLoadModuleEx+0x80
25 00000049`7a07eb50 00007ff6`fc5f025f     SymStore!SymCommonGetFileInformation+0x13f
26 00000049`7a07f290 00007ff9`1b09257d     SymStore!Add1ChildThread+0x57f
27 00000049`7a07fed0 00007ff9`1cf6af28     KERNEL32!BaseThreadInitThunk+0x1d
28 00000049`7a07ff00 00000000`00000000     ntdll!RtlUserThreadStart+0x28

0:004> gn
Critical error detected c0000374
(96c.542c): Break instruction exception - code 80000003 (first chance)
ntdll!RtlReportCriticalFailure+0x56:
00007ff9`1d01ca32 cc              int     3
0:004> k
 # Child-SP          RetAddr               Call Site
00 00000049`7a079ae0 00007ff9`1d025b0a     ntdll!RtlReportCriticalFailure+0x56
01 00000049`7a079bd0 00007ff9`1d025dea     ntdll!RtlpHeapHandleError+0x12
02 00000049`7a079c00 00007ff9`1d031de5     ntdll!RtlpHpHeapHandleError+0x7a
03 00000049`7a079c30 00007ff9`1d025a93     ntdll!RtlpLogHeapFailure+0x45
04 00000049`7a079c60 00007ff9`1cf50793     ntdll!RtlpAnalyzeHeapFailure+0x30f
05 00000049`7a079cc0 00007ff9`1cf4d239     ntdll!RtlpAllocateHeap+0x1663
06 00000049`7a079f20 00007ff9`1d014772     ntdll!RtlpAllocateHeapInternal+0x6c9
07 00000049`7a07a020 00007ff9`1cfce0da     ntdll!RtlDebugAllocateHeap+0x102
08 00000049`7a07a0c0 00007ff9`1cf4d239     ntdll!RtlpAllocateHeap+0x7efaa
09 00000049`7a07a320 00007ff9`1a5e33ae     ntdll!RtlpAllocateHeapInternal+0x6c9
0a 00000049`7a07a420 00007ff9`1a6220cb     ucrtbase!_calloc_base+0x4e
0b 00000049`7a07a450 00007ff9`1a622051     ucrtbase!_vcrt_getptd_noexit+0x63
0c 00000049`7a07a480 00007ff9`1a6148da     ucrtbase!_vcrt_getptd+0x9
0d 00000049`7a07a4b0 00007ff9`1cfb51df     ucrtbase!_CxxFrameHandler3+0x2a
0e 00000049`7a07a500 00007ff9`1cf2e866     ntdll!RtlpExecuteHandlerForException+0xf
0f 00000049`7a07a530 00007ff9`1cfb41ce     ntdll!RtlDispatchException+0x286
10 00000049`7a07ac80 00007ff8`632435d5     ntdll!KiUserExceptionDispatch+0x2e
11 00000049`7a07ba50 00007ff8`6325d3fd     dbghelp!DBI1::addFileInfo+0x125
12 00000049`7a07bac0 00007ff8`632597ff     dbghelp!Mod1::processC13+0x7ad
13 00000049`7a07bd70 00007ff8`6325ad7b     dbghelp!Mod1::fProcessSyms+0x11ff
14 00000049`7a07be30 00007ff8`632504e7     dbghelp!Mod1::fUpdateSyms+0x1b
15 00000049`7a07be60 00007ff8`6323638f     dbghelp!Mod1::Close+0x207
16 00000049`7a07bee0 00007ff8`63221dd4     dbghelp!PortablePDB::ConvertPortablePDB+0x3df
17 00000049`7a07c820 00007ff8`632211cd     dbghelp!PDB1::OpenPortablePDB+0x84
18 00000049`7a07c860 00007ff8`63221585     dbghelp!PDB1::OpenEx2W+0x5ed
19 00000049`7a07c920 00007ff8`6323273b     dbghelp!PDB::OpenEx2W+0x35
1a 00000049`7a07c970 00007ff8`632898c7     dbghelp!PDBCommon::Open2W+0x2b
1b 00000049`7a07c9c0 00007ff8`63289729     dbghelp!CDiaDataSource::loadDataFromPdbHelper+0x57
1c 00000049`7a07ca10 00007ff8`6332a2fd     dbghelp!CDiaDataSource::loadDataFromPdb+0x19
1d 00000049`7a07ca50 00007ff8`63336e9a     dbghelp!DiaOpenPdbEx+0x201
1e 00000049`7a07cb00 00007ff8`633363d9     dbghelp!diaOpenPdb+0x56
1f 00000049`7a07cb60 00007ff8`6333336b     dbghelp!diaLocatePdb+0x17d
20 00000049`7a07d8c0 00007ff8`6335837a     dbghelp!diaGetPdb+0x29f
21 00000049`7a07db90 00007ff8`6335cce8     dbghelp!GetDebugData+0x2de
22 00000049`7a07de20 00007ff8`6335c99b     dbghelp!modloadWorker+0x2e0
23 00000049`7a07e320 00007ff8`63358ff4     dbghelp!modload+0xff
24 00000049`7a07e5c0 00007ff8`63352b30     dbghelp!LoadModule+0x608
25 00000049`7a07eae0 00007ff6`fc5f58cf     dbghelp!SymLoadModuleEx+0x80
26 00000049`7a07eb50 00007ff6`fc5f025f     SymStore!SymCommonGetFileInformation+0x13f
27 00000049`7a07f290 00007ff9`1b09257d     SymStore!Add1ChildThread+0x57f
28 00000049`7a07fed0 00007ff9`1cf6af28     KERNEL32!BaseThreadInitThunk+0x1d
29 00000049`7a07ff00 00000000`00000000     ntdll!RtlUserThreadStart+0x28

0:004> gn
(96c.542c): Unknown exception - code c0000374 (first chance)
(96c.542c): Unknown exception - code c0000374 (!!! second chance !!!)
ntdll!RtlReportFatalFailure+0x9:
00007ff9`1d01caa9 eb00            jmp     ntdll!RtlReportFatalFailure+0xb (00007ff9`1d01caab)
0:004> k
 # Child-SP          RetAddr               Call Site
00 00000049`7a079a90 00007ff9`1d01ca73     ntdll!RtlReportFatalFailure+0x9
01 00000049`7a079ae0 00007ff9`1d025b0a     ntdll!RtlReportCriticalFailure+0x97
02 00000049`7a079bd0 00007ff9`1d025dea     ntdll!RtlpHeapHandleError+0x12
03 00000049`7a079c00 00007ff9`1d031de5     ntdll!RtlpHpHeapHandleError+0x7a
04 00000049`7a079c30 00007ff9`1d025a93     ntdll!RtlpLogHeapFailure+0x45
05 00000049`7a079c60 00007ff9`1cf50793     ntdll!RtlpAnalyzeHeapFailure+0x30f
06 00000049`7a079cc0 00007ff9`1cf4d239     ntdll!RtlpAllocateHeap+0x1663
07 00000049`7a079f20 00007ff9`1d014772     ntdll!RtlpAllocateHeapInternal+0x6c9
08 00000049`7a07a020 00007ff9`1cfce0da     ntdll!RtlDebugAllocateHeap+0x102
09 00000049`7a07a0c0 00007ff9`1cf4d239     ntdll!RtlpAllocateHeap+0x7efaa
0a 00000049`7a07a320 00007ff9`1a5e33ae     ntdll!RtlpAllocateHeapInternal+0x6c9
0b 00000049`7a07a420 00007ff9`1a6220cb     ucrtbase!_calloc_base+0x4e
0c 00000049`7a07a450 00007ff9`1a622051     ucrtbase!_vcrt_getptd_noexit+0x63
0d 00000049`7a07a480 00007ff9`1a6148da     ucrtbase!_vcrt_getptd+0x9
0e 00000049`7a07a4b0 00007ff9`1cfb51df     ucrtbase!_CxxFrameHandler3+0x2a
0f 00000049`7a07a500 00007ff9`1cf2e866     ntdll!RtlpExecuteHandlerForException+0xf
10 00000049`7a07a530 00007ff9`1cfb41ce     ntdll!RtlDispatchException+0x286
11 00000049`7a07ac80 00007ff8`632435d5     ntdll!KiUserExceptionDispatch+0x2e
12 00000049`7a07ba50 00007ff8`6325d3fd     dbghelp!DBI1::addFileInfo+0x125
13 00000049`7a07bac0 00007ff8`632597ff     dbghelp!Mod1::processC13+0x7ad
14 00000049`7a07bd70 00007ff8`6325ad7b     dbghelp!Mod1::fProcessSyms+0x11ff
15 00000049`7a07be30 00007ff8`632504e7     dbghelp!Mod1::fUpdateSyms+0x1b
16 00000049`7a07be60 00007ff8`6323638f     dbghelp!Mod1::Close+0x207
17 00000049`7a07bee0 00007ff8`63221dd4     dbghelp!PortablePDB::ConvertPortablePDB+0x3df
18 00000049`7a07c820 00007ff8`632211cd     dbghelp!PDB1::OpenPortablePDB+0x84
19 00000049`7a07c860 00007ff8`63221585     dbghelp!PDB1::OpenEx2W+0x5ed
1a 00000049`7a07c920 00007ff8`6323273b     dbghelp!PDB::OpenEx2W+0x35
1b 00000049`7a07c970 00007ff8`632898c7     dbghelp!PDBCommon::Open2W+0x2b
1c 00000049`7a07c9c0 00007ff8`63289729     dbghelp!CDiaDataSource::loadDataFromPdbHelper+0x57
1d 00000049`7a07ca10 00007ff8`6332a2fd     dbghelp!CDiaDataSource::loadDataFromPdb+0x19
1e 00000049`7a07ca50 00007ff8`63336e9a     dbghelp!DiaOpenPdbEx+0x201
1f 00000049`7a07cb00 00007ff8`633363d9     dbghelp!diaOpenPdb+0x56
20 00000049`7a07cb60 00007ff8`6333336b     dbghelp!diaLocatePdb+0x17d
21 00000049`7a07d8c0 00007ff8`6335837a     dbghelp!diaGetPdb+0x29f
22 00000049`7a07db90 00007ff8`6335cce8     dbghelp!GetDebugData+0x2de
23 00000049`7a07de20 00007ff8`6335c99b     dbghelp!modloadWorker+0x2e0
24 00000049`7a07e320 00007ff8`63358ff4     dbghelp!modload+0xff
25 00000049`7a07e5c0 00007ff8`63352b30     dbghelp!LoadModule+0x608
26 00000049`7a07eae0 00007ff6`fc5f58cf     dbghelp!SymLoadModuleEx+0x80
27 00000049`7a07eb50 00007ff6`fc5f025f     SymStore!SymCommonGetFileInformation+0x13f
28 00000049`7a07f290 00007ff9`1b09257d     SymStore!Add1ChildThread+0x57f
29 00000049`7a07fed0 00007ff9`1cf6af28     KERNEL32!BaseThreadInitThunk+0x1d
2a 00000049`7a07ff00 00000000`00000000     ntdll!RtlUserThreadStart+0x28

0:004> gn
ntdll!NtWaitForWorkViaWorkerFactory+0x14:
00007ff9`1cfb3d84 c3              ret

Regression?

Yes, these errors started to occur after switching from .NET Framework to .NET8 and changing the project type to SDK.

Known Workarounds

Use full .pdb format instead of portable.

Other information

At the moment it seems that the problem only occurs on large files, but it is confirmed consistently.

[LunarWhisper]

@tommcdon , @grantri FYI

[dotnet-policy-service]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

[grantri]

@LunarWhisper - please provide a repro. At a minimum you can provide the PDB that causes the crash. I suspect this the same issue, there is just confusion about what constitutes a "file" from the PDB's perspective versus the user's perspective. But without repro step or a repro PDB, we can only speculate.

[LunarWhisper]

@LunarWhisper - please provide a repro. At a minimum you can provide the PDB that causes the crash. I suspect this the same issue, there is just confusion about what constitutes a "file" from the PDB's perspective versus the user's perspective. But without repro step or a repro PDB, we can only speculate.

@grantri, I'm afraid this will require signing an NDA.