Fix | Allow SqlBulkCopy to operate on hidden columns

[edwardneal]

Description

This deals with an older bug: if we try to use SqlBulkCopy to import data into a SQL Server table with hidden columns, we can't use those hidden columns as destinations - SqlBulkCopy won't acknowledge that the column exists.

The root cause of this is that we discover the columns in the target table by running the command below and looking at the TDS-level metadata:

SET FMTONLY ON;
SELECT * FROM {table}
SET FMTONLY OFF;

Hidden columns don't appear when we run a SELECT * query. This PR overcomes the problem by querying sys.all_columns for the column list, building a SELECT query dynamically.

It's worth noting that this PR doesn't result in a successful bulk copy. As far as I can tell, the only way to create a hidden column is to create a temporal table - specific columns can be marked GENERATED ALWAYS AS ROW START HIDDEN. Trying to insert data into these columns will result in a SqlException containing error 13536 from SQL Server:

Cannot insert an explicit value into a GENERATED ALWAYS column in table '{database}.{schema}.{table}'.

This is a better failure mode than the existing behaviour (an InvalidOperationException) though - it makes it clear that SqlBulkCopy found the column and that SQL Server is rejecting it.

Issues

Fixes #1854.

Testing

New unit test added, all SqlBulkCopy unit tests pass.

The original issue mentioned that running DROP PERIOD FOR SYSTEM_TIME would disable versioning and leave the columns hidden. I wasn't able to reproduce this - once the table has been altered, the columns are no longer hidden.

[benrr101]

After working on sqltoolsservice and DSCT, I learned how it's pretty much always frowned upon to use * for "official" queries. So, I'm kinda surprised we were using * to discover the columns in a table.

A few changes that could be made, but I won't hold up the PR over them.

[benrr101]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[benrr101]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[benrr101]

@edwardneal so looking at the failures, I think the issue is because in main we've dropped DataTestUtility.GetUniqueNameForSqlServer and split it into GetShortName and GetLongName. Even though it's not a conflict, I guess we're doing the merge before running the CI builds? Anyways, take a look at the notes for those methods, and we'll get the PR going.

[edwardneal]

Thanks @benrr101, I've just merged to main.

[benrr101]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[benrr101]

@edwardneal looks like there's a test related to hidden columns failing :(

Failed Microsoft.Data.SqlClient.ManualTesting.Tests.SqlBulkCopyTests.HiddenTargetColumn.WriteToServer_CopyToHiddenTargetColumn_ThrowsSqlException [1 s]
EXEC : error Message:  [/mnt/vss/_work/1/s/build.proj]
     Assert.StartsWith() Failure: String start does not match
  String:         "Cannot insert an explicit value into a GE"···
  Expected start: "Cannot insert an explicit value into a GE"···
    Stack Trace:
       at Microsoft.Data.SqlClient.ManualTesting.Tests.SqlBulkCopyTests.HiddenTargetColumn.WriteToServer_CopyToHiddenTargetColumn_ThrowsSqlException() in /_/src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/SqlBulkCopyTest/HiddenTargetColumn.cs:line 59
     at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
     at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

[edwardneal]

That's frustrating, but thanks. Our test case is covered by the SqlException.Number assertion on the previous line, so I've simplified the comparison.

It's odd to see this fail though - I've tested against SQL 2022, 2025 and Azure...

[paulmedynski]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[benrr101]

Looks good to me! Thanks for hanging in there on the test failures!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (37d8a3f) to head (0444e6e).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #3590       +/-   ##
==========================================
- Coverage   66.13%       0   -66.14%     
==========================================
  Files         276       0      -276     
  Lines       60765       0    -60765     
==========================================
- Hits        40184       0    -40184     
+ Misses      20581       0    -20581     

Flag	Coverage Δ
addons	?
netcore	?
netfx	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.