Provider compliance/test suite for ADO.NET

[roji]

To promote standard behavior across ADO.NET providers, it would be great if a test suite existed. Ideally all provider-independent tests would be extracted from SqlClient and made available via a nuget. This would be similar to what the EF Core team has done.

[saurabh500]

@roji
Are you suggesting that we add tests based on System.Data.Common and these tests are used to test any ADO.Net providers by instantiating the actual provider objects?
Can you point me to the EF tests that you are referring to?

[roji]

@saurabh500 yes, that's right. Providers such as Npgsql would extend test base classes provided by you, thus running your standardized test suite. Your tests would instantiate ADO.NET objects via the provider's DbProviderFactory and make sure that the provider's DbConnection, DbCommand, etc. behave as they should.

For the EFCore provider test suite, see Microsoft.EntityFrameworkCore.Specification.Tests and Microsoft.EntityFrameworkCore.Relational.Specification.Tests. The source code is here and here respectively. A good example is QueryTestBase, which runs various query scenarios and makes sure the results come back as expected.

It's true that ADO.NET is a mature technology which already has well-established providers, so such a "specification" test suite might be less valuable than in EFCore, which is totally new technology where providers can use the specification while implementing. But even with ADO.NET it would be very useful to automatically find divergences in provider behavior, leading to either standardized behavior or at least explicit documentation of those divergences.

[roji]

Note that Npgsql contains many tests which are completely generic - i.e. aren't specific to Npgsql/PostgreSQL in any way. It is exactly this kind of tests that belong in a provider-independent specification test library.

[saurabh500]

@roji do you want to grab this task?

[roji]

@saurabh500 I have a lot on my plate, but I can try to spend some time on this (not right away though). Also, I'm pretty PostgreSQL-centric and will definitely need help with making sure things are portable.

[saurabh500]

Sounds good.

[roji]

Just to say that I'm unlikely to find any time to work on this soon... Hopefully Microsoft (or someone else) can pick this up.

[divega]

cc @bricelam what do you think about refactoring the Microsoft.Data.Sqlite test to follow the test spec pattern we already use for EF Core provider tests?

[bricelam]

They weren't written with that purpose in mind, but it could be a good starting point since they're very functional in nature.

[divega]

@bricelam I think that is probably true for (at least some of) the SqlClient tests.

[bgrainger]

This is something I've been interested in having for a while, too (as the author of MySqlConnector). I got tired of manually writing test cases to check compatibility across different drivers, so I created an automated test harness. 🙂

A WIP rough draft is here: https://github.com/mysql-net/AdoNetApiTest

I bootstrapped the prototype with @bricelam's SqliteCommand tests (which saved a lot of time, thanks!).

Inspired by http://seriot.ch/json/parsing.html, it dumps output like the following. (Take the colours in the cells with a grain of salt; I haven't investigated them all to ensure that they're not caused by a bug in the test harness itself.)

[roji]

@bgrainger that looks like a wonderful start - I'm excited to see this.

Although I expect to be too busy until end of December, I definitely look forward to contributing to this and evolving it. In the meantime, please take a look at the Npgsql tests - a lot of them are exactly the kind of generic, functional tests that would fit perfectly into this.

One more comment - while it's really useful for this to exist as a self-contained project that includes connectors to specific ADO.NET providers, it's also important that projects like Npgsql and MySqlConnector be able to use this in their solutions, and to override certain things. In other words, there will need to be a generic abstract nuget that specific ADO.NET drivers can consume, extending the abstract test suite. Some tests may be overridden (so everything needs to be virtual), some would be ignored/skipped (not relevant for that provider), etc. etc.

[bgrainger]

Thanks; that's a great suggestion (and I hadn't thought that far ahead yet...). Created mysql-net/AdoNetApiTest#3 to discuss it (without hijacking this issue).