Skip to content

Conversation

@bob80905
Copy link
Collaborator

@bob80905 bob80905 commented Sep 9, 2025

This PR focuses on allowing dxc to use the existing infrastructure to use a DxcDllExtValidationLoader object, and load an external dxil.dll with it, via the environment variable. The validate*.test was added to verify that the dll is indeed being loaded and used for external validation, which causes a failure, since the dll's validator version is older than the required minimum validator version that the metadata indicates.

Fixes #7527

@github-actions
Copy link
Contributor

github-actions bot commented Sep 13, 2025

✅ With the latest revision this PR passed the C/C++ code formatter.

@bob80905 bob80905 changed the title Make DXC use new DxcDllExtValidationLoader to load external validator before validation. Make DXC and DXV use new DxcDllExtValidationLoader to load external validator before validation. Sep 16, 2025
@bob80905 bob80905 force-pushed the add_extvalsupport_class_to_val_infra branch from 6c69325 to 78cace7 Compare September 16, 2025 23:24
@bob80905 bob80905 marked this pull request as ready for review September 17, 2025 18:21
Comment on lines 9 to 16
UINT32 newArgCount = ArgCount + 1;
std::vector<LPCWSTR> newArgs;
newArgs.reserve(newArgCount);

// Copy existing args
for (unsigned int i = 0; i < ArgCount; ++i) {
newArgs.push_back(pArguments[i]);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think rewriting this as below is equally performant and certainly cleaner

Suggested change
UINT32 newArgCount = ArgCount + 1;
std::vector<LPCWSTR> newArgs;
newArgs.reserve(newArgCount);
// Copy existing args
for (unsigned int i = 0; i < ArgCount; ++i) {
newArgs.push_back(pArguments[i]);
}
std::vector<LPCWSTR> newArgs = {pArguments, pArguments + ArgCount};

Comment on lines 115 to 116
if (pResult == nullptr)
return E_POINTER;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
if (pResult == nullptr)
return E_POINTER;
if (!pResult)
return E_POINTER;

hr = evh->QueryInterface(riid, reinterpret_cast<void **>(pResult));
return hr;

} else if (clsid == CLSID_DxcValidator) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: you can delete the else here. Might even be worth moving the entire if to the top of this block since its quite short comparted to the other one

IUnknown **pResult) {
if (DxilExtValSupport.IsEnabled() && clsid == CLSID_DxcValidator)
return DxilExtValSupport.CreateInstance2(pMalloc, clsid, riid, pResult);
if (pResult == nullptr)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
if (pResult == nullptr)
if (!pResult)

Comment on lines 913 to 914
if (DXC_FAILED(ValHR)) {
if (SUCCEEDED(pCompileResult->QueryInterface(&pResult)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it looks like these can be a single if statement

@github-project-automation github-project-automation bot moved this from New to In progress in HLSL Roadmap Sep 19, 2025
// validation scenario, so there is no point in also running
// the internal validator. This wrapper wraps both the
// IDxcCompiler and IDxcCompiler3 interfaces.
class ExternalValidationHelper : public IDxcCompiler, public IDxcCompiler3 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For one, you'll need to wrap all the interfaces that the compiler object implements so you don't lose the object wrapping for any of them.

But also, IDxcCompiler and IDxcCompiler3 should be incompatible to derive from and implement in the same class due to the nature of COM not supporting separate overloaded methods of the same name. This requires a separate object to implement the other interfaces.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So to be clear, are you suggesting one parent class, ExternalValidationHelper, that implements 4 interfaces and has 4 wrapper member objects, that each implement one interface: IDxcCompiler, IDxcCompiler2, IDxcCompiler3, and IDxcValidator? Your comment above, "This code should all be inside the compiler object wrapper.", makes me think that this class should also implement the IDxcValidator interface, is that correct?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But also, IDxcCompiler and IDxcCompiler3 should be incompatible to derive from and implement in the same class due to the nature of COM not supporting separate overloaded methods of the same name.

It's lucky that IDxcCompiler and IDxcCompiler3 don't actually have any identical methods in them so that this isn't a problem.

IDxcCompiler::Compile takes 10 args, while IDxcCompiler3::Compile takes 7
Preprocess only exists on IDxcCompiler
IDxcCompiler::Disassemble takes 2 args while IDxcCompiler3::Disassemble take 3.

If you only try and implement one of these when you derive from both interfaces you'll get a warning about hiding the base class version.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. I thought it was not allowed, hence the hoops we jumped through multiple times for multiple implementations of the compiler object supporting both interfaces to avoid deriving from both on the same implementation class.

In any case the first point stands, that there are more interfaces to support than these two.

You'll want to derive from each of these:

  • IDxcCompiler2
  • IDxcCompiler3
  • IDxcLangExtensions3
  • IDxcContainerEvent
  • IDxcVersionInfo3
  • IDxcVersionInfo2 (or IDxcVersionInfo if SUPPORT_QUERY_GIT_COMMIT_INFO not defined)

And no need to derive from IDxcCompiler when deriving from IDxcCompiler2, since IDxcCompiler2 already derives from IDxcCompiler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation in this PR will cause failures if any of the tests attempt to use these interfaces. I don't think we need to wrap additional ones for the sake of it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like Damyan said, I think I'll worry about deriving from each of those in a future PR.

Copy link
Contributor

@tex3d tex3d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are quite a few issues, and as I tried to write something that could express the exact logic necessary for the arguments, I started rewriting the ExternalValidationHelper implementation to also fix many of the other issues. I put a branch up, so here's the link to the commit:

tex3d@08bacdb

See the commit for the general list of issues this rewrite fixes.

@bob80905 bob80905 force-pushed the add_extvalsupport_class_to_val_infra branch from c6ac790 to eb69e26 Compare September 24, 2025 05:36
Copy link
Member

@damyanp damyanp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this still looks good to me. It's very hard for me to review, because the rebase means that I can't easily tell what changed since my last review. Since we squash all our commits into main there's no real benefit to rebasing rather than merging with main.

Comment on lines 51 to 54
if (Result)
return Result->QueryInterface(Riid, ValResult);
else
return E_FAIL;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
if (Result)
return Result->QueryInterface(Riid, ValResult);
else
return E_FAIL;
if (!Result)
return E_FAIL;
return Result->QueryInterface(Riid, ValResult);

Copy link
Contributor

@alsepkow alsepkow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submitting a block of comments while I continue to review.

if (Result)
return Result->QueryInterface(Riid, ValResult);
else
return E_FAIL;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest returning E_POINTER as I think that's what this HRESULT is intended to reflect?
That is, we expect that the caller gives us a valid IDxcOperationResult*.


// No validation needed; just set the result and return.
if (!NeedsValidation)
return UseResult(CompileResult);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The UseResult name seems misleading to me? It really looks like it's just a glorified QueryInterface call?

I think it would be cleaner to

  1. Remove the UseResult lambda
  2. Add a check to return E_INVALID_ARG if CompileResult is an invalid pointer at the start of doValidation.
  3. Update line 59 to 'return CompileResult->QueryInterface(Riid, ValResuilt);
  4. Same update to line 67.
  5. Same update for line 83.
  6. Same update on line 87.

Comment on lines 104 to 105
if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor,
&ValidatorVersionMinor)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor,
&ValidatorVersionMinor)) {
if (FAILED(ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor,
&ValidatorVersionMinor))) {

#endif
!Opts.DumpDependencies && !Opts.VerifyDiagnostics &&
Opts.Preprocess.empty();
bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel;
const bool ProduceFullContainer = ProduceDxModule && !Opts.CodeGenHighLevel;

return DoBasicQueryInterface<IDxcCompiler, IDxcCompiler2, IDxcCompiler3>(
NewWrapper.p, Iid, ResultObject);
} catch (...) {
return E_FAIL;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the exception expected here would be due to an allocation failure.
So would ERROR_OUTOFMEMORY be a more appropriate HRESULT?

if (ValidatorVersionInfo->GetVersion(&ValidatorVersionMajor,
&ValidatorVersionMinor)) {
DXASSERT(false, "Failed to get validator version");
return E_FAIL;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I like to try and avoid E_FAIL because its pretty generic.
There is an 'ERROR_VERSION_PARSE_ERROR' you could use here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna stick to E_FAIL since there is no definition for that Error code in non-windows.
Same applies to ERROR_OUTOFMEMORY

// The ExtValidationArgHelper class helps manage arguments for external
// validation, as well as perform the validation step if needed.
class ExtValidationArgHelper {
std::vector<std::wstring> ArgStorage;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we generally do all publics first and then all privates second?
Should we move the member variables to the private section?

// IDxcValidator interface to validate a successful compilation result, when
// validation would normally be performed.
class ExternalValidationCompiler : public IDxcCompiler2, public IDxcCompiler3 {
private:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: public first for consistency with the rest of this file

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the reason this broke that consistency is because there's a special rule in linux / macos builds on initialization order, and the Compiler object can't be initialized after the m_pAlloc field, which was originally taken care of in the previous private block. So I will leave this as is.

Copy link
Member

@damyanp damyanp Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's a special rule in linux / macos builds on initialization order

There's not a special rule, it's just that MSVC doesn't warn if you mismatch the order that things appear in the constructor's initialization list with the order that they are actually initialized (ie the order they appear in the class itself).

If you wanted to put the private block at the end then you'd just need to update the constructor to list m_pMalloc first (since it'd be listed first in the class as the field is added by the DXC_MICROCOM_TM_REF_FIELDS macro).

Or you could have your private block at the end look like this and not need to modify the constructor:

private:
  CComPtr<IDxcValidator> Validator;
  IID CompilerIID;
  CComPtr<IUnknown> Compiler;
  DXC_MICROCOM_TM_REF_FIELDS()

CComPtr<IDxcBlob> CompiledBlob;
IFR(CompileResult->GetResult(&CompiledBlob));

// If no compiled blob; just return the compile result.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: comment doesn't seem accurate. We're returning the HRESULT of QI. That doesn't seem like a compile result?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This language is a bit overloaded, I meant return through an out-parameter, by placing the result into ValResult.
I can rewrite this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main thing that threw me off is that you wouldn't generally expect a QI call to do anything other than get you a pointer to the requested interface. The fact that the private implementation of QI is doing more than that is a little weird.

// Return the validation result if it failed.
HRESULT HR;
IFR(TempValidationResult->GetStatus(&HR));
if (FAILED(HR)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see uses of DXC_FAILED and FAILED. Hadn't seen DXC_FAILED before but it looks like they do the same thing.
We should just use one.

// function is deferred to whatever object was chosen to be pCompiler,
// which must implement the IDxcCompiler interface. External validation
// will only take place if the DXC_DXIL_DLL_PATH env var is set
// correctly.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this comment on this one compile call? Why not put a comment where you initialize the DxcDllExtValidationLoader in dxc::main instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the Compile command also is responsible for validation. I think this call site is appropriate because at this point, different outcomes may happen. IMO it's less clear on DxcDllExtValidationLoader initialization, since there's no compilation or validation happening then and there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, another nit I'll let go.

I guess my point was:

  • This isn't the only call that could lead to external validation in DxcContext
  • This isn't the code scope where that knowledge is needed or relevant - in fact, as I pointed out elsewhere, DxcContext doesn't need to be hard-coded to use DxcDllExtValidationLoader in the first place.

I suggested main because that's the top-level place where options are parsed and this is a bit like an undocumented option for the tool which dxcSupport.initialize() is going to read from the environment at that point.

private:
DxcOpts &m_Opts;
SpecificDllLoader &m_dxcSupport;
DxcDllExtValidationLoader &m_dxcSupport;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we use DllLoader instead? Isn't that the point of DllLoader as a base class?

Suggested change
DxcDllExtValidationLoader &m_dxcSupport;
DllLoader &m_dxcSupport;

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of DllLoader is to provide abstraction for an object that is capable of creating instances of objects from a dll.
However, in the specific cases of dxc and dxv, we know ahead of time that in that context, we don't need any abstraction. dxc and dxv should both be using the more specific object type, because we know that they may both load from an external dll if the environment variable exists.
The DllLoader class was created for cases where there's delegation, and where it isn't clear whether a DxcDllExtValidationLoader instance was passed or whether something else might be passed. But we can constrain dxc and dxv to be DxcDllExtValidationLoader's only.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get that, and it's just a nit, so I'm fine with this.

Just to be clear why I thought we should use DllLoader, and consider it a nit:

This DxcContext is in a library and can be linked into and used from different top-level tool driver executables. If one of those doesn't need to use DxcDllExtValidationLoader and implements its own main to initialize the context, it doesn't seem right (architecturally) to force it to use DxcDllExtValidationLoader since DxcContext only needs to know about the DllLoader interface.

I'm fine with it as-is particularly because we don't plan to expand the set of tools using DxcContext. We are more likely to deprecate/remove dxl instead, which, IIRC, is the only other tools that uses this.

Copy link
Contributor

@tex3d tex3d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed we regressed the -external and -external-fn dxc options, and need a change to DxcDllExtValidationLoader to support this.

@bob80905 bob80905 force-pushed the add_extvalsupport_class_to_val_infra branch from f69c61f to 5a3821f Compare October 15, 2025 21:29
Copy link
Contributor

@tex3d tex3d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but would like to see the fragile __FILE__ reference from the test replaced with a reference to a file in the test data location which is required to be set for the test environment.

template <typename T> CComPtr<T> cast() const {
CComPtr<T> Result;
if (Compiler)
Compiler->QueryInterface(__uuidof(T), reinterpret_cast<void **>(&Result));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but why doesn't Result = Compiler; work? CComPtr will correctly call the QueryInterface for you upon assignment.

Even if you want to manually write the QueryInterface, you can use Compiler->QueryInterface(IID_PPV_ARGS(&Result));, or Compiler.QueryInterface(&Result); (CComPtr's templated QueryInterface).

VERIFY_IS_TRUE(ExtSupportBogus.dxilDllFailedToLoad());

// 3. Test with a valid path to this file in the environment variable
std::filesystem::path p = std::filesystem::absolute(__FILE__);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This presumes the test has access to this source file via the same path. That seems quite fragile, since that's not a requirement of the tests.

We do have a reliable way to access a file that must exist in the test environment: hlsl_test::GetPathToHlslDataFile(fileName) where fileName is the path to a test file relative to the tools/clang/test/HLSL path, for instance, you could use another file referenced in ValidationTest.cpp: L"..\\DXILValidation\\hs_signatures.hlsl" (through the CompileFile() function).

@bob80905 bob80905 merged commit 715dacc into microsoft:main Oct 27, 2025
12 checks passed
@github-project-automation github-project-automation bot moved this from In progress to Done in HLSL Roadmap Oct 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Replace DxilDllSupport in dxc.cpp and in tests

5 participants