Implement SQLite ADO.NET schema collections

[alaricd]

Why

Fixes #35917.

The common schema collections (MetaDataCollections, DataSourceInformation, DataTypes, Restrictions, ReservedWords) are part of the documented ADO.NET provider contract, which describes them as implemented by each managed provider:
https://learn.microsoft.com/dotnet/framework/data/adonet/common-schema-collections

ORMs that ship their own SQLite dialect (EF Core, NHibernate) don't need the provider to describe itself, so they aren't the right test for this. Code written against DbProviderFactory / DbConnection does need it. Without DataSourceInformation, a generic consumer has no supported way to discover SQLite's parameter marker format, identifier pattern or version, so it has to special-case SQLite by name — the provider-specific code the ADO.NET abstraction exists to remove. Background on the pattern:
https://dev.to/pengdows/stop-hardcoding-your-net-data-provider-4jmo

Compatibility

This is additive. GetSchema does not currently return these collections, so no existing caller's behavior changes.

Scope

If the full change is more than the team wants to maintain, I'm happy to reduce it to only the common collections and drop the provider-specific Tables/Columns/Indexes.

References:

• GetSchema and schema collections
• Common schema collections
• SQLite PRAGMA documentation
• SQLite parameter syntax
• SQLite tokenizer and identifier requirements
• SQLite 3.39 release notes

What changed

• Implemented DataSourceInformation, DataTypes, MetaDataCollections, and Restrictions.
• Added provider-specific Tables, Columns, and Indexes collections.
• If the provider-specific collections are outside the intended scope, I can remove them and leave the four standard collections needed by pengdows.crud: DataSourceInformation, DataTypes, MetaDataCollections, and Restrictions.
• Included attached databases with correct schema-qualified metadata.
• Added normalized SQLite versions, prefix-free parameter-marker formatting, corrected regex capture groups, standard schema/table/nullability values, and SQLite internal-table filtering.
• Added version-gated pragma_table_list support for virtual tables and shadow-table exclusion on SQLite 3.37 and later, with a sqlite_master fallback for older SQLite builds. The fallback cannot distinguish virtual-table shadow tables.
• Included virtual-table columns when the module is available; broken views and unavailable virtual-table modules are skipped per object rather than aborting the collection.
• Reduced regular-table metadata collection to one query per schema and used the hidden schema argument for table-valued pragmas.

TDD and validation

Tests cover all implemented collections, attached databases, same-name tables across schemas, parameter-marker consumption, normalized versions, regex capture groups, standard metadata values, escaped sqlite_ filtering, broken views, FTS5 virtual tables, shadow tables, empty databases, and version-specific join operators.

Validation:

• Build completed with 0 warnings and 0 errors.
• Full Microsoft.Data.Sqlite.Tests assembly: 756 passed, 7 expected skips, 0 failed.

[alaricd]

@dotnet-policy-service agree company="Pengdows LLC"

[Copilot]

🟡 Changes recommended

Parameter formatting and column metadata contain correctness issues, while broad exception handling can hide operational failures.

4 open findings

• Return prefixed parameter marker format · New
• Include generated columns in schema metadata · New
• Recognize implicit non-nullability of integer primary keys · New
• Do not suppress unexpected SQLite schema errors · New

What changed in this PR

Implements ADO.NET schema discovery for Microsoft.Data.Sqlite.

Changes:

• Adds common provider metadata collections.
• Adds table, column, and index discovery across attached databases.
• Adds comprehensive schema tests.

File	Description
src/​Microsoft.Data.Sqlite.Core/​SqliteConnection.cs	Implements schema metadata collection.
test/​Microsoft.Data.Sqlite.Tests/​SqliteConnectionTest.cs	Tests metadata and SQLite schema behavior.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Several metadata values and edge cases still produce incorrect or incomplete schema results.

2 open findings

• Misidentifies composite INTEGER primary key columns as non-null · New
• Broken views abort schema retrieval instead of being skipped · New

4 resolved since last review

• Do not suppress unexpected SQLite schema errors
• Recognize implicit non-nullability of integer primary keys
• Include generated columns in schema metadata
• Return prefixed parameter marker format

Previously missed (3)

In code that hasn't changed since last review

Drops SQLite version revision during normalization

src/​Microsoft.Data.Sqlite.Core/​SqliteConnection.cs:862

The normalized value drops Version.Revision, so four-component SQLite patch releases such as 3.8.7.1 and 3.8.7.2 both become 03.008.0007. Since Microsoft.Data.Sqlite.Core supports custom and system native libraries, this violates the common-schema requirement that normalized versions remain string-comparable across provider versions. Include a fixed-width revision component (using zero when absent) and cover a four-component version.

Marks dynamically typed REAL columns as fixed length

src/​Microsoft.Data.Sqlite.Core/​SqliteConnection.cs:913

IsFixedLength is set to true for REAL, but SQLite uses dynamic typing: a column declared REAL can still store variable-length text or blobs. The common schema contract defines this flag in terms of columns created by DDL, so generic metadata consumers would incorrectly assume fixed-length storage here.

Reports false signedness for nonnumeric TEXT and BLOB types

src/​Microsoft.Data.Sqlite.Core/​SqliteConnection.cs:915

These rows return false for IsUnsigned, but the common DataTypes contract requires DBNull.Value when signedness is not applicable. TEXT and BLOB are nonnumeric, so reporting false misrepresents them as signed types to generic consumers.

🧠 Review effort: Balanced

[AndriySvyryd]

Thanks for your contribution!

[bricelam]

It looks like NHibernate assumes a standard set of restrictions for these:

Collection	Restrictions
Tables	catalog, schema, tableName
Columns	catalog, schema, tableName, columnName
Indexes	catalog, schema, tablename, indexName, null

At the very least, maybe we should ignore extra restrictions by removing the check above:

https://github.com/pengdows/efcore/blob/41071837fb8c2fe835fb0a0916292d2d5f2f02b3/src/Microsoft.Data.Sqlite.Core/SqliteConnection.cs#L810-L813

[alaricd]

I agree that the ADO.NET schema contract should be the source of truth here. SQLite currently reports zero supported restrictions, so rejecting supplied restrictions is intentional. NHibernate’s expected restriction layout may be useful for future compatibility, but I don’t think we should silently ignore values or advertise restrictions we do not implement. We can add specific restrictions later if we implement their filtering semantics and update the metadata accordingly.

[bricelam]

rejecting supplied restrictions is intentional

Yes, it was my intention at the time. 😉 But for the sake of pragmatism, it might be worth reconsidering. I think most providers ignore "extra" restrictions--you can append all kinds of nonsense onto the end of the array.

[alaricd]

I tested this against the advertised ADO.NET metadata: Tables, Columns, and Indexes all report zero supported restrictions, and tests verify that supplying restrictions is rejected rather than silently ignored. I'm keeping that behavior because silently discarding caller-supplied filters would be misleading while the provider does not implement restriction semantics. If compatibility with a specific consumer becomes necessary, we can add explicit restrictions and filtering in a follow-up.

[bricelam]

It might be confusing to return all of these. The subsequent pragmas only return items for the default database (main). To get items from other attached databases, you need to call them with the database name: PRAGMA attached-db.table_list

[bricelam]

See also #23587 for A LOT more work that needs to be done in this area.

[alaricd]

This is already handled in the current implementation. The table-valued pragmas are invoked with the database name as the schema argument (for example, pragma_table_info(m.name, $schema) and pragma_index_list(m.name, $schema)), and the table-list path filters by m.schema. The existing GetSchema_pragma_collections_include_attached_databases test covers attached database tables, columns, and indexes.

[alaricd]

Agreed that #23587 represents broader schema-discovery work. The attached-database behavior in this PR is covered and tested; I'm leaving unrelated schema-surface expansion for that follow-up.

[bricelam]

Suggested change

	=> sqliteVersion >= new Version(3, 39) ? 15 : 3;
	=> sqliteVersion >= new Version(3, 39)
	? (int)(SupportedJoinOperators.Inner | SupportedJoinOperators.LeftOuter
	| SupportedJoinOperators.RightOuter | SupportedJoinOperators.FullOuter)
	: (int)(SupportedJoinOperators.Inner | SupportedJoinOperators.LeftOuter);

[alaricd]

The current values are the same flag combinations represented by the suggested enum: 3 for INNER | LEFT OUTER before SQLite 3.39, and 15 for all four join operators from SQLite 3.39 onward. The DataTypes metadata column is an integer, so the behavior is covered by the existing version-specific tests.

[bricelam]

Just calling out my preference of using the actual enum over "magic" numbers.

[alaricd]

Agreed. I updated the follow-up PR (#39219) in commit 5e7247def6 to use System.Data.Common.SupportedJoinOperators flags instead of raw numeric values, while casting to the required integer for the schema column. The existing version-specific tests now assert the enum values.

[alaricd]

Follow-up validation completed in commit 264984bb84:

• Added four-component normalized-version coverage and preserved Version.Revision (zero-filled when absent).
• Corrected REAL.IsFixedLength to false.
• Corrected TEXT and BLOB IsUnsigned to DBNull.Value, as signedness is not applicable.
• Added restriction-behavior tests for all advertised zero-restriction collections.

The core project builds cleanly and git diff --check passes. The focused test project could not be restored in this environment because the configured NuGet feeds were unavailable.