Skip to content

Commit

Permalink
Expose metadata to encode/tryDecode (#106) resolves #102
Browse files Browse the repository at this point in the history
  • Loading branch information
bartelink authored Mar 5, 2019
1 parent 365f8ef commit 0d7401d
Show file tree
Hide file tree
Showing 17 changed files with 190 additions and 161 deletions.
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,14 @@ _NB at the present time, this project does not adhere strictly to Semantic Versi
## [Unreleased]

### Added

- Provide capability to access `Metadata` and `EventNumber`/`Index` re [#102](https://github.com/jet/equinox/issues/102)

### Changed

- Make `caching` non-optional in CosmosStreamResolver; add `NoCaching` cache mode for `Equinox.Cosmos` [#104](https://github.com/jet/equinox/issues/104) @jakzale
- Make `caching` non-optional in `CosmosStreamResolver`; add `NoCaching` cache mode for `Equinox.Cosmos` [#104](https://github.com/jet/equinox/issues/104) @jakzale
- Reorder `caching` and `access` in `GesStreamResolver` to match `CosmosStreamResolver` [#107](https://github.com/jet/equinox/issues/107)
- Renamespaced and separated `Equinox.Codec` APIs to separate `Newtonsoft.Json` and custom `encode`/`tryDecode` approaches [#102](https://github.com/jet/equinox/issues/102) (in preparation for [#79](https://github.com/jet/equinox/issues/79))

### Removed
### Fixed
Expand Down
10 changes: 4 additions & 6 deletions DOCUMENTATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -687,7 +687,7 @@ The higher level APIs (i.e. not `Core`), as demonstrated by the `dotnet new` tem

```fsharp
open Equinox.Cosmos.Core.Events
open Equinox.Cosmos.Core
// open MyCodecs.Json // example of using specific codec which can yield UTF-8 byte arrays from a type using `Json.toBytes` via Fleece or similar
type EventData with
Expand Down Expand Up @@ -783,15 +783,12 @@ It should be noted that the `Equinox.Projection.Kafka` batteries included projec

# Roadmap

# Very likely to happen and/or people looking at it:

- Extend samples and templates; see [#57](https://github.com/jet/equinox/issues/57)
- Enable snapshots to be stored outside of the main collection in `Equinox.Cosmos` see [#61](https://github.com/jet/equinox/issues/61)

# Things that are incomplete and/or require work

This is a very loose laundry list of items that have occurred to us to do, given infinite time. No conclusions of likelihood of starting, finishing, or even committing to adding a feature should be inferred, but most represent things that would be likely to be accepted into the codebase (please raise Issues first though ;) ).

- Extend samples and templates; see [#57](https://github.com/jet/equinox/issues/57)

## Wouldn't it be nice - `Equinox.EventStore`:

EventStore, and it's Store adapter is the most proven and is pretty feature rich relative to the need of consumers to date. Some things remain though:
Expand All @@ -807,6 +804,7 @@ EventStore, and it's Store adapter is the most proven and is pretty feature rich

## Wouldn't it be nice - `Equinox.Cosmos`:

- Enable snapshots to be stored outside of the main collection in `Equinox.Cosmos` see [#61](https://github.com/jet/equinox/issues/61)
- Multiple writers support for `u`nfolds (at present a `sync` completely replaces the unfolds in the Tip; this will be extended by having the stored proc maintain the union of the unfolds in play (both for semi-related services and for blue/green deploy scenarios); TBD how we decide when a union that's no longer in use gets removed)
- performance improvements in loading logic
- `_etag`-based consistency checks?
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ While Equinox is implemented in F#, and F# is a great fit for writing event-sour
- Designed not to invade application code; Domain tests can be written directly against your models without any need to involve or understand Equinox assemblies or constructs as part of writing those tests.
- Extracted from working software; currently used for all data storage within Jet's API gateway and Cart processing.
- Significant test coverage for core facilities, and with baseline and specific tests per Storage system and a comprehensive test and benchmarking story
- Encoding of events via `Equinox.UnionCodec` provides for pluggable encoding of events based on either:
- Encoding of events via `Equinox.Codec` provides for pluggable encoding of events based on either:
- a [versionable convention-based approach](https://eiriktsarpalis.wordpress.com/2018/10/30/a-contract-pattern-for-schemaless-datastores/) (using `Typeshape`'s `UnionContractEncoder` under the covers), providing for serializer-agnostic schema evolution with minimal boilerplate
- an explicitly coded pair of `encode` and `tryDecode` functions for when you need to customize
- Independent of the store used, Equinox provides for caching using the .NET `MemoryCache` to minimize roundtrips, latency and bandwidth / Request Charges by maintaining the folded state, without necessitating making the Domain Model folded state serializable
Expand Down Expand Up @@ -203,8 +203,8 @@ The components within this repository are delivered as a series of multi-targete

- [![NuGet](https://img.shields.io/nuget/v/Equinox.svg)](https://www.nuget.org/packages/Equinox/) `Equinox[.Stream]`: Store-agnostic decision flow runner that manages the optimistic concurrency protocol. ([depends](https://www.fuget.org/packages/Equinox) on `Serilog` (but no specific Serilog sinks, i.e. you configure to emit to `NLog` etc))
- [![Codec NuGet](https://img.shields.io/nuget/v/Equinox.Codec.svg)](https://www.nuget.org/packages/Equinox.Codec/) `Equinox.Codec`: [a scheme for the serializing Events modelled as an F# Discriminated Union](https://eiriktsarpalis.wordpress.com/2018/10/30/a-contract-pattern-for-schemaless-datastores/) ([depends](https://www.fuget.org/packages/Equinox.Codec) on `TypeShape 6.*`, `Newtonsoft.Json >= 11.0.2` but can support any serializer) with the following capabilities:
- independent of any specific serializer
- allows tagging of F# Discriminated Union cases in a versionable manner with low-dependency `DataMember(Name=` tags using [TypeShape](https://github.com/eiriktsarpalis/TypeShape)'s [`UnionContractEncoder`](https://github.com/eiriktsarpalis/TypeShape/blob/master/tests/TypeShape.Tests/UnionContractTests.fs)
- `Equinox.Codec.JsonNet.JsonUtf8`: allows tagging of F# Discriminated Union cases in a versionable manner with low-dependency `DataMember(Name=` tags using [TypeShape](https://github.com/eiriktsarpalis/TypeShape)'s [`UnionContractEncoder`](https://github.com/eiriktsarpalis/TypeShape/blob/master/tests/TypeShape.Tests/UnionContractTests.fs)
- `Equinox.Codec.JsonUtf8`: independent of any specific serializer; enables plugging in a serializer and/or Union Encoder of your choice

### Store libraries

Expand Down
4 changes: 2 additions & 2 deletions samples/Infrastructure/Services.fs
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ open Microsoft.Extensions.DependencyInjection
open System

let serializationSettings = Newtonsoft.Json.Converters.FSharp.Settings.CreateCorrect()
let genCodec<'Union when 'Union :> TypeShape.UnionContract.IUnionContract>() = Equinox.UnionCodec.JsonUtf8.Create<'Union>(serializationSettings)
let genCodec<'Union when 'Union :> TypeShape.UnionContract.IUnionContract>() = Equinox.Codec.JsonNet.JsonUtf8.Create<'Union>(serializationSettings)

type StreamResolver(storage) =
member __.Resolve
( codec : Equinox.UnionCodec.IUnionEncoder<'event,byte[]>,
( codec : Equinox.Codec.IUnionEncoder<'event,byte[]>,
fold: ('state -> 'event seq -> 'state),
initial: 'state,
snapshot: (('event -> bool) * ('state -> 'event))) =
Expand Down
4 changes: 2 additions & 2 deletions samples/Store/Integration/CodecIntegration.fs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ let serializationSettings =
Newtonsoft.Json.Converters.FSharp.OptionConverter() |])

let genCodec<'Union when 'Union :> TypeShape.UnionContract.IUnionContract>() =
Equinox.UnionCodec.JsonUtf8.Create<'Union>(serializationSettings)
Equinox.Codec.JsonNet.JsonUtf8.Create<'Union>(serializationSettings)

type EventWithId = { id : CartId } // where CartId uses FSharp.UMX

Expand Down Expand Up @@ -51,6 +51,6 @@ let codec = genCodec<SimpleDu>()
[<AutoData(MaxTest=100)>]
let ``Can roundtrip, rendering correctly`` (x: SimpleDu) =
let serialized = codec.Encode x
render x =! if serialized.payload = null then null else System.Text.Encoding.UTF8.GetString(serialized.payload)
render x =! if serialized.Data = null then null else System.Text.Encoding.UTF8.GetString(serialized.Data)
let deserialized = codec.TryDecode serialized |> Option.get
deserialized =! x
2 changes: 1 addition & 1 deletion samples/Store/Integration/EventStoreIntegration.fs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ open Equinox.EventStore
open System

let serializationSettings = Newtonsoft.Json.Converters.FSharp.Settings.CreateCorrect()
let genCodec<'Union when 'Union :> TypeShape.UnionContract.IUnionContract>() = Equinox.UnionCodec.JsonUtf8.Create<'Union>(serializationSettings)
let genCodec<'Union when 'Union :> TypeShape.UnionContract.IUnionContract>() = Equinox.Codec.JsonNet.JsonUtf8.Create<'Union>(serializationSettings)

/// Connect with Gossip based cluster discovery using the default Commercial edition Manager port config
/// Such a config can be simulated on a single node with zero config via the EventStore OSS package:-
Expand Down
2 changes: 1 addition & 1 deletion samples/Tutorial/Todo.fsx
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ module Store =
let cacheStrategy = CachingStrategy.SlidingWindow (cache, TimeSpan.FromMinutes 20.)

module TodosCategory =
let codec = Equinox.UnionCodec.JsonUtf8.Create<Event>(Newtonsoft.Json.JsonSerializerSettings())
let codec = Equinox.Codec.JsonUtf8.Create<Event>(Newtonsoft.Json.JsonSerializerSettings())
let access = Equinox.Cosmos.AccessStrategy.Snapshot (isOrigin,compact)
let resolve = CosmosResolver(Store.store, codec, fold, initial, Store.cacheStrategy, access=access).Resolve

Expand Down
123 changes: 123 additions & 0 deletions src/Equinox.Codec/Codec.fs
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
namespace Equinox.Codec

/// Common form for either a Domain Event or an Unfolded Event
type IEvent<'Format> =
/// The Event Type, used to drive deserialization
abstract member EventType : string
/// Event body, as UTF-8 encoded json ready to be injected into the Store
abstract member Data : 'Format
/// Optional metadata (null, or same as Data, not written if missing)
abstract member Meta : 'Format

/// Defines a contract interpreter for a Discriminated Union representing a set of events borne by a stream
type IUnionEncoder<'Union, 'Format> =
/// Encodes a union instance into a decoded representation
abstract Encode : value:'Union -> IEvent<'Format>
/// Decodes a formatted representation into a union instance. Does not throw exception on format mismatches
abstract TryDecode : encoded:IEvent<'Format> -> 'Union option

/// Provides Codecs that render to a UTF-8 array suitable for storage in EventStore or CosmosDb based on explicit functions you supply
/// i.e., with using conventions / Type Shapes / Reflection or specific Json processing libraries - see Equinox.Codec.JsonNet for batteries-included Coding/Decoding
type JsonUtf8 =

/// <summary>
/// Generate a codec suitable for use with <c>Equinox.EventStore</c> or <c>Equinox.Cosmos</c>,
/// using the supplied pair of <c>encode</c> and <c>tryDecode</code> functions. </summary>
/// <param name="encode">Maps a 'Union to an Event Type Name with UTF-8 arrays representing the `Data` and `Metadata`.</param>
/// <param name="tryDecode">Attempts to map from an Event Type Name and UTF-8 arrays representing the `Data` and `Metadata` to a 'Union case, or None if not mappable.</param>
// Leaving this private until someone actually asks for this (IME, while many systems have some code touching the metadata, it tends to fall into disuse)
static member private Create<'Union>(encode : 'Union -> string * byte[] * byte[], tryDecode : string * byte[] * byte[] -> 'Union option)
: IUnionEncoder<'Union,byte[]> =
{ new IUnionEncoder<'Union, byte[]> with
member __.Encode e =
let eventType, payload, metadata = encode e
{ new IEvent<_> with
member __.EventType = eventType
member __.Data = payload
member __.Meta = metadata }
member __.TryDecode ee =
tryDecode (ee.EventType, ee.Data, ee.Meta) }

/// <summary>
/// Generate a codec suitable for use with <c>Equinox.EventStore</c> or <c>Equinox.Cosmos</c>,
/// using the supplied pair of <c>encode</c> and <c>tryDecode</code> functions. </summary>
/// <param name="encode">Maps a 'Union to an Event Type Name and a UTF-8 array representing the `Data`.</param>
/// <param name="tryDecode">Attempts to map an Event Type Name and a UTF-8 `Data` array to a 'Union case, or None if not mappable.</param>
static member Create<'Union>(encode : 'Union -> string * byte[], tryDecode : string * byte[] -> 'Union option)
: IUnionEncoder<'Union,byte[]> =
let encode' value = let c, d = encode value in c, d, null
let tryDecode' (et,d,_md) = tryDecode (et, d)
JsonUtf8.Create(encode', tryDecode')

namespace Equinox.Codec.Core

/// Represents a Domain Event or Unfold, together with it's Index in the event sequence
// Included here to enable extraction of this ancillary information (by downcasting IEvent in one's IUnionEncoder.TryDecode implementation)
// in the corner cases where this coupling is absolutely definitely better than all other approaches
type IIndexedEvent<'Format> =
inherit Equinox.Codec.IEvent<'Format>
/// The index into the event sequence of this event
abstract member Index : int64
/// Indicates this is not a Domain Event, but actually an Unfolded Event based on the state inferred from the events up to `Index`
abstract member IsUnfold: bool

/// An Event about to be written, see IEvent for further information
// Storage implementations couple to IEvent - this is included to facilitate eventual consistency across the .Core per-Store programming models
// (at the time of writing, Equinox.Cosmos.Core is the only such API)
type EventData<'Format> =
{ eventType : string; data : 'Format; meta : 'Format }
interface Equinox.Codec.IEvent<'Format> with member __.EventType = __.eventType member __.Data = __.data member __.Meta = __.meta
type EventData =
static member Create(eventType, data, ?meta) = { eventType = eventType; data = data; meta = defaultArg meta null}

namespace Equinox.Codec.JsonNet

open Newtonsoft.Json
open System.IO
open System.Runtime.InteropServices

/// Newtonsoft.Json implementation of TypeShape.UnionContractEncoder's IEncoder that encodes direct to a UTF-8 Buffer
type Utf8BytesUnionEncoder(settings : JsonSerializerSettings) =
let serializer = JsonSerializer.Create(settings)
interface TypeShape.UnionContract.IEncoder<byte[]> with
member __.Empty = Unchecked.defaultof<_>
member __.Encode (value : 'T) =
use ms = new MemoryStream()
( use jsonWriter = new JsonTextWriter(new StreamWriter(ms))
serializer.Serialize(jsonWriter, value, typeof<'T>))
ms.ToArray()
member __.Decode(json : byte[]) =
use ms = new MemoryStream(json)
use jsonReader = new JsonTextReader(new StreamReader(ms))
serializer.Deserialize<'T>(jsonReader)

/// Provides Codecs that render to a UTF-8 array suitable for storage in EventStore or CosmosDb based on explicit functions you supply using `Newtonsoft.Json` and
/// TypeShape.UnionContract.UnionContractEncoder - if you need full control and/or have have your own codecs, see Equinox.Codec.JsonUtf8 instead
type JsonUtf8 =

/// <summary>
/// Generate a codec suitable for use with <c>Equinox.EventStore</c> or <c>Equinox.Cosmos</c>,
/// using the supplied `Newtonsoft.Json` <c>settings</c>.
/// The Event Type Names are inferred based on either explicit `DataMember(Name=` Attributes,
/// or (if unspecified) the Discriminated Union Case Name
/// The Union must be tagged with `interface TypeShape.UnionContract.IUnionContract` to signify this scheme applies.
/// See https://github.com/eiriktsarpalis/TypeShape/blob/master/tests/TypeShape.Tests/UnionContractTests.fs for example usage.</summary>
/// <param name="settings">Configuration to be used by the underlying <c>Newtonsoft.Json</c> Serializer when encoding/decoding.</param>
/// <param name="requireRecordFields">Fail encoder generation if union cases contain fields that are not F# records. Defaults to <c>false</c>.</param>
/// <param name="allowNullaryCases">Fail encoder generation if union contains nullary cases. Defaults to <c>true</c>.</param>
static member Create<'Union when 'Union :> TypeShape.UnionContract.IUnionContract>
( settings,
[<Optional;DefaultParameterValue(null)>]?requireRecordFields,
[<Optional;DefaultParameterValue(null)>]?allowNullaryCases)
: Equinox.Codec.IUnionEncoder<'Union,byte[]> =
let dataCodec =
TypeShape.UnionContract.UnionContractEncoder.Create<'Union,byte[]>(
new Utf8BytesUnionEncoder(settings),
?requireRecordFields=requireRecordFields,
?allowNullaryCases=allowNullaryCases)
{ new Equinox.Codec.IUnionEncoder<'Union,byte[]> with
member __.Encode value =
let enc = dataCodec.Encode value
Equinox.Codec.Core.EventData.Create(enc.CaseName, enc.Payload) :> _
member __.TryDecode encoded =
dataCodec.TryDecode { CaseName = encoded.EventType; Payload = encoded.Data } }
4 changes: 2 additions & 2 deletions src/Equinox.Codec/Equinox.Codec.fsproj
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@
</PropertyGroup>

<ItemGroup>
<Compile Include="UnionCodec.fs" />
<Compile Include="Codec.fs" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0-beta2-18618-05" PrivateAssets="All"/>
<PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0-beta2-18618-05" PrivateAssets="All" />
<PackageReference Include="MinVer" Version="1.0.0-rc.1" PrivateAssets="All" />

<PackageReference Include="FSharp.Core" Version="3.1.2.5" Condition=" '$(TargetFramework)' == 'net461' " />
Expand Down
Loading

0 comments on commit 0d7401d

Please sign in to comment.