The big 6.0.0 release thread

[adrianhall]

Hey folks - kicking off a big 6.0.0 thread.

Some of the things happening:

• Dropping support for Xamarin (.NET Standard 2.0 focused stuff) to concentrate on .NET 6
• Dropping support for JObject based tables (generics only)
• ... which allows us to support LiteDb and Entity Framework Core based tables
• ... which allows us to support more types during the conversion
• Re-work the operations queue, delta-token store, etc. to be part of the offline store
• ... which allows us to wrap the write to the store and operations queue into a transaction
• ... which makes the client interaction more "clean"
• Move over to System.Text.Json
• ... which makes the pipeline more memory efficient and better performing

Work has already started in my own repository, but I'll be merging changes as they are done into the release/6.0.0 branch.

What else do you want?

[adrianhall]

Update 2/24/2023

I've started work on the new client

• The DatasyncClient will now build an IHttpClientFactory - there is a DatasyncHttpClientFactory that builds clients similar to the old ServiceHttpClient, but this allows me to integrate the IHttpClientFactory that Blazor uses, which means I can support Blazor WASM and Blazor Server.
• I've split a lot of the client abstractions out into a separate package. This will help when I get to writing the EF Core SQLite driver and LiteDb driver since a lot of the work will be common.
• Wrote tests for all the work thus far.
• Set up build pipelines for the new stuff. This means I can test with "real code" by utilizing the ADO artifacts feed in visual studio.

Next, I'm going to start introducing (again) all the old tests for the ASP.NET Core. Also, note that .NET 8 Preview 1 came out, and that is LTS. I'm going to keep an eye on that and move the version to 8.0.0 and re-compile to .NET 8 once it comes out (at that point, it should be an easy swap).

[ideas2appsaustralia]

Adrian integrating with blazor is a great idea. When would you think 6.0 will be ready to test?

[adrianhall]

Once I have "critical mass" of code (which isn't there yet). My intent is to get authentication and remote tables done first, then do offline tables, then do synchronization. I'll mention what is ready once I have the code checked in. You'll have to download the code and build it yourself, however.

There won't be too much work done over the next couple of weeks because I'm going on vacation to get some sun.

[dylan-mills]

Do you think critical mass will be in a few weeks, a month or 6 months? Just asking to help plan our app since it looks like 6.0 may have some breaking changes in it.

[adrianhall]

Good question - probably closer to 6 months, but I doubt it will be that long.

[adrianhall]

Update 3/15/2023

I'm going to do a re-spin after discussing the project with a couple of architects here at Microsoft. Basically, we'll be structuring things like this:

1. Microsoft.Datasync.Client.Http - the HTTP code for implementing a CRUDL REST API Service Client
2. Microsoft.Datasync.Client.Query - the codes necessary to handle LINQ queries and turn them into OData or whatever else we want.
3. Microsoft.Datasync.Client.Abstractions - things common to all the SQLite stores (among others)
4. Microsoft.Datasync.Client - the codes necessary to implement a RemoteTable and the centralized clients.
5. Microsoft.Datasync.Client.OfflineStore - an individual offline store implementation.

This, plus a few other changes I've got in mind, will enable us to support gRPC and GraphQL, more types of offline store. Obviously, the re-structuring is more than just projects - the code fundamentally changes as well. I'll be working on these changes this week and hope to have something checked in by the end of the week.

[amertlich]

Are these only client side changes and nothing server side?
What do you anticipate the effort being to migrate an existing app to this new architecture?
We're just about finished migrating from 4 -> 5 which was probably few months of effort (we're not as fast as you, Adrian ;) and granted this wasn't the only change we've been making...). Are you anticipating 5 -> 6 to be a similar amount of change? How long before you anticipate 6 becoming the stable release?

[adrianhall]

Correct - I'm not expecting anything server side.

I'll try and make the changes to the actual used API surface very small, but the infrastructure should be much more performant. At the end of it, my aim is to do a version bump, set up the new offline store, and then do a wipe/synchronize of the offline store to get it into the new format.

Some of the changes I'm proposing right now are breaking, though. I'm attempting to minimize those.

[adrianhall]

Update 3/31/2023

I've started doing some of the structural work in the main branch this week. Specifically, there is a new interface for the delta-token store, and the implementation of the delta-token store has moved into the SQLite implementation. If the store implements IDeltaTokenStoreProvider, then the delta-token store implementation is retrieved from the store rather than using a default implementation. Eventually, the default implementation will "go away" and you will have to implement this yourself.

How does this affect you? In the next release, if you are implementing your own offline store, then you also need to implement a delta-token store at some point in the next 6-ish months. You can do it any time, but I would hold off for now, or just copy over the SQLiteDeltaTokenStore version.

Why am I doing this? It's a longer term plan to get all the offline store stuff encapsulated within the offline store - including queuing and synchronization. The eventual aim is to wrap the queue stuff into the offline store as well. At that point, you can wrap the queue and the record update into a single transaction, allowing for faster queue operations. The design for this phase is still being done, but the concepts work out. This will allow me to support different (non-SQL) stores and maybe even move to 6.0.0 with no breaking changes.

[ideas2appsaustralia]

Adrian will there be ability to put images/videos into an offline store which can also be synced to suitable azure store? Just thinking about apps where clients want to record a photo as a record of work etc?

[adrianhall]

I'm not intending on it. It's a reasonable ask, but I want to think about the best way to do this. Could you answer the following questions?

1. How big are the photos?
2. How often / how many photos would be in a single request?
3. Is it one-way (mobile to cloud) or two-way (will get downloaded to other devices)?
4. Will it be synchronized with a database change?
5. Is Azure Blob Storage the only end storage?
6. How will you control the size of the store holding the photos? How do you think of expiry of photos, for example?

[amertlich]

We've currently implemented our own solution for this, but happy to provide some feedback. If you come up with something that meets our needs too, we'd perhaps adopt it.

1. We currently limit attachments/photos to be under 20mb/attachment although have had requests to even allow videos to be attached.
2. Number of attachments per synchronized record varies on use case, but there can be several attachments/photos (even dozens) associated with an individual record.
3. It's generally mobile to cloud. We don't automatically synchronize down attachments associated with records -- the user has to initiate a manual pull or explicitly state they want the attachments fully synchronized down.
4. The push is synchronized with any database change (although it happens after all sync table changes are pushed. And can be delayed as the record data will always take precedence over an attachment upload). We do thumbnail the attachment and store it with the sync'ed record base64 encoded. The pull by default does not pull down associated attachments.
5. Right now we are using Azure Blob Storage. Alternative hosting options would be nice, but not required for us.
6. Our users can choose to disassociate/remove/delete attachments from records individually. They can also delete entire "projects" that have several records/attachments. When records are deleted, the associated attachments are also deleted. All attachments for a customer's project are grouped in sub-containers in Azure storage so when the project is deleted or a subscription has not be paid for a sufficient time we can clean out and delete all storage associated to a customers project.

Let me know if you have any questions!