Want to consider populating $all position for stream reads/subscriptions. gRPC & non-transaction only.

[timothycoleman]

Added: Populate $all position for stream reads/subscriptions/persistent subscriptions. gRPC only. Non-transaction events only.

Currently the $all position (commit and prepare pair) is only provided to the client during $all reads/subscriptions, and not during stream reads/subscriptions. However, the user-facing datastructure does contain fields for this information.

This PR considers the possibility of populating the $all position for stream reads/subscriptions with the following caveats

• only for gRPC clients (the tcp wire datastructures do not have fields for this information)
• only for events that are not written in explicit transactions (we cannot efficiently determine the commit position for events in transactions)

todo: discuss pros (this is asked for from time to time. a bit odd that the fields are present but not currently populated) and cons (potential confusion with respect to transactions). does it need a server capability added? does it want backporting?

Client impact:

• dotnet, java, go, rust: most probably no changes necessary
• node: will not be broken by this change, but needs some work to plumb the data through to the user. can be written in a non-breaking way

Closes kurrent-io/KurrentDB-Client-Dotnet#211.

[PhotoAtomic]

Hello, any update on this PR? Can I help in any way to speed up the merge into master?
I've just tested it and it is EXACTLY what I was waiting for! It is like Christmas but earlier :D !
I cant wait to have this in an official release.
Thanks @timothycoleman !

[PhotoAtomic]

Performing some more test have unfortunately highlighted that there is still something to fix:
Everything works fine if I read from the stream where i've wrote the event in, in this case ResolvedEvent.Event.Position and ResolvedEvent.OriginalPosition correctly returns the Position of the event

Instead, if I read the same event from a projection, ResolvedEvent.Event.Position and ResolvedEvent.OriginalPosition are the same value, and none of this is the value of the Poisition of the inserted even in its stream.

To my understanding ResolvedEvent.OriginalPosition should be the Poistion in the projection but ResolvedEvent.Event.Position really should be the Position of the Event in its own Stream, so the exact same value of the Position from when I read from the stream i've written it into.

[TestMethod]
        public async Task RetrievedEventShouldProvideTheirRealPosition()
        {
            string eventDBConnectionString = @"esdb://admin:changeit@localhost:2113?tls=false&tlsVerifyCert=false";

            var settings = EventStoreClientSettings
                .Create(eventDBConnectionString);
            var client = new EventStoreClient(settings);


            var evt = new TestEvent
            {
                EntityId = Guid.NewGuid().ToString("N"),
                ImportantData = "I wrote my first event!"
            };

            var eventData = new EventData(
                Uuid.NewUuid(),
                "TestEvent",
                JsonSerializer.SerializeToUtf8Bytes(evt)
            );


            var appendResult = await client.AppendToStreamAsync(
                "some-stream",
                StreamState.Any,
                new[] { eventData }                
            );

            await Task.Delay(1000);

            var result = client.ReadStreamAsync(
                Direction.Forwards,
                "some-stream",                
                StreamPosition.Start,
                resolveLinkTos: true);

            var events = await result.ToListAsync();

            Assert.AreEqual(events.Last().Event.Position, appendResult.LogPosition);
            Assert.AreNotEqual(events.Last().Event.Position, Position.End);            

            var resultFromProjection = client.ReadStreamAsync(
                Direction.Forwards,
                "$category-some",
                StreamPosition.Start,
                resolveLinkTos: true);

            //Was: var eventsFromProjection = await result.ToListAsync();  there is a copy-paste bug here, the correct line is below, thanks Timothy!
            var eventsFromProjection = await resultFromProjection.ToListAsync();

            Assert.AreEqual(eventsFromProjection.Last().Event.Position, appendResult.LogPosition);
            Assert.AreEqual(eventsFromProjection.Last().Event.Position, events.Last().Event.Position);
            Assert.AreNotEqual(eventsFromProjection.Last().Event.Position, Position.End);
        }

We are so close!!!! How can I know the Position of the original event? Again, "All" seems to know it correctly, so there should be a way.

[timothycoleman]

Thanks for the repro, did you mean to read $ce-some instead of $category-some? $ce has links to events ($>) that the server can resolve for you and (ought to) work as you're hoping. $category is a bit different, it links to streams ($@) rather than events, and the server doesn't resolve them to events, so in your test you're testing the location of the link rather than the location of the testevent

[PhotoAtomic]

Hello Timothy! Thanks for the fast reply!
Yes I would like to use the $ce-some as you correctly guessed, but I had a stupid bug in my test and I was tricked into thinking that the correct one was $category, my bad!
Thanks to your suggestion, I've spotted my error and I'm pleased to say that it works perfectly!
I'm super happy! thanks!

Do you think it will be possible to have this fantastic fix in an official version soon?
Thanks again for your help!

[CristinaBenaglio]

Hello Timothy!
Also my dev team will really benefit from this fix, it will help a lot to improve the overall performance of our innovative product, and I can't wait to have this in an official release of EventStoreDB!
Many thanks 😊

[timothycoleman]

@ylorph ^

[PhotoAtomic]

Wow! I'm so happy to seeing progress on this task and now that also the test are in place, the merge looks really close! thanks again to Timothy for the work, you really can't imagine how much this will improve my application, I can't wait to have this in the main main branch and packed in a nice container image.

If I've well understood there is still to wait for @thefringeninja (who have done some related work in PR#2816 and @shaan1337 review.
Thanks in advance for your help guys!
Again, if I can help in any way, maybe some test or anything, just let me know!

[timothycoleman]

thanks @PhotoAtomic and you're very welcome.

If you could test it out in your application (if you haven't ready) and let us know if you have any problems that would be great.

Apart from that the ticket is just going through our normal review and then QA process (which is quite busy at the moment with other things too). It'll be in the next release (22.6) which will likely be out in June or July

[PhotoAtomic]

My unit tests (on the branch) so far indicates that everything is fine and works accordingly my expectation.
I've now to reimplement a "library" layer in order to take advantage of the new feature, but that's my system business.
It looks that from the EventStoreDB side everything works as expected!
Mote on the topic when I'll have the library reimplemented :)