Support type provider invalidation or compiler reset during watched file recompilation

[zanaptak]

Description

Currently, watched file recompilation of a file with a type provider fails to update the provided type definitions when the underlying data changes. The type definition is generated only on initial compile and remains stuck in its initial state for the duration of the webpack-dev-server session. This results in compiler errors during recompile in watch mode, and requires a stop and restart to pick up the changes.

Any chance this could be made to work, either by the formal TP invalidation mechanism, or else some kind of reset option on the internal compiler to recompile from scratch instead of whatever internal TP state it is currently holding on to?

Repro code

Here is a repro project, using a modified Fable.JsonProvider with invalidation and logging added.

Repro steps:

Clone repo: https://github.com/zanaptak/TestFableTypeProvider.git
Run: dotnet build ./TypeProvider (because App.fsproj has a file path reference to the debug version of this assembly)
Run: npm install
Run (Fable 2): npm start
Run (Fable 3): dotnet tool restore
Run (Fable 3): dotnet fable watch App --run webpack-dev-server
Open App folder in VS Code (recommend disabling background services in ionide settings and restarting first, to reduce log noise)
Observe: IDE and Fable compile both work, no type errors

In schema.json, change foo1 to foo2 (this triggers invalidation of the TP)
In App.fs, if error not already appeared, make an inconsequential edit to trigger a type check
Observe: IDE should recognize the TP invalidation, regen the type definition, and show error on the old json.foo1 property name

In App.fs, change json.foo1 to json.foo2, and save
Observe: IDE is happy with new property name
Observe: Fable watched file recompilation fails. The new code change is picked up, but the TP was not refreshed, and the new property name does not exist on the stale type definition.

Check the TypeProvider.log file in the App dir and observe differences in how both processes behave:
Observe: IDE supports invalidation, resulting in incrementing TP instances [0], [1]… in the log.
Observe: Fable uses a single TP instance [0]. It appears to call into the TP when the recompile happens. We have no caching in the TP so we return an updated type definition, but Fable seems to be ignoring it and keeping the stale type.

Expected and actual results

Expected: A change to TP underlying data recompiles successfully in watch mode.

Actual: Recompilation fails, requires restart of webpack-dev-server.

Related information

• Fable version: 2.10.2 3.0.0-nagareyama-rc-004
• Operating system: Windows 10

[alfonsogarciacaro]

Hmm, this is going to be tricky. Actually there's almost no code in Fable to deal with Type Providers, that's mostly done by the FSharp.Compiler.Services. I did try your repo with Ionide and the IDE couldn't detect changes in the JSON schema either, not sure how other IDEs to it.

[zanaptak]

VS Code + Ionide seems to be unreliable but it does work. I just tried again, it took a few reloads and then sitting idle for 5 mins to recognize the project, but then worked correctly as described in the repro steps. I assume it is an IDE issue and not really a good representation of the actual type provider mechanism. VS 2019 works without any issue.

After some investigation I believe it is possible to address this issue in service_slim.fs. I will do some more testing and update with more details.

[zanaptak]

This appears to be handled in the "official" service.fs by subscribing to a CcuData.InvalidateEvent and triggering an InvalidateConfiguration function to reinitialize a new IncrementalBuilder whenever a type provider invalidates.

I used that general idea in a proof of concept service_slim.fs and it works. The type provider schema invalidation is recognized by dotnet fable watch, and it correctly recompiles code with the updated TP data.

Key points are:

Enable TP invalidation:

https://github.com/zanaptak/TestFableTypeProvider/blob/a4aa6e0ad743897368e8190dec015cd1b3855d8f/service_slim.fs#L60

Subscribe to invalidation event:

https://github.com/zanaptak/TestFableTypeProvider/blob/a4aa6e0ad743897368e8190dec015cd1b3855d8f/service_slim.fs#L77

Reset internal compiler state when invalidation happens:

https://github.com/zanaptak/TestFableTypeProvider/blob/a4aa6e0ad743897368e8190dec015cd1b3855d8f/service_slim.fs#L89-L99

[ncave]

@zanaptak Looks great, one minor question is does compilerState need to be an option.
Please feel free to PR it, or I can merge it myself if you prefer.

[zanaptak]

@ncave no the current implementation is unsafe with option .Value references. I think it needs a top level "get or add" cache mechanism at the public member entry points and then pass retrieved compilerState to the private members (similar to tcState param) to also avoid race condition on compiler state if invalidation happens during a compile process.

I can make a PR, should it go into https://github.com/ncave/fsharp repo?

[ncave]

@zanaptak Yes, you can target the service_slim branch in https://github.com/ncave/fsharp repo.

[zanaptak]

@ncave PR opened: ncave/fsharp#6

[ncave]

@zanaptak @alfonsogarciacaro Merged in service_slim.