Subject Indexing and Ordering Internals

[Zetanova]

I would have following questions that I can't answer after reading the docs and discussions.
The QA would be essential to use Nats/Jetstream as an EventSourcing store.

How does Nats/JetStream index subjects for filtered consumers?
How does Nats/JetStream perform with a lot of unique subjects names over a long period?
Does Nats provide the same ordering guaranty for messages
publish on a.s1.eventType1 and a.s1.eventType2 as only published to a.s1 ?

The unique AggregateRoot-Id count can reach view millions of entries per year.
Of course it depends on the business domain.
But in most cases a single AR is only used/appended over a short period of time (days)
and for a good EventSourcing store the index/subject count should not effect the performance of the "tail" stream.

[derekcollison]

How does Nats/JetStream index subjects for filtered consumers?

We have meta state that allows us to index efficiently by sequence number and by subject. We will continue to improve upon this but walking a filtered set of messages with the modern server should be efficient today.

How does Nats/JetStream perform with a lot of unique subjects names over a long period?

Currently the subject addressing layer to a stream takes more memory the more unique subjects that you have. We are working to make this more memory friendly in future releases and plan to have some improvements in 2.10. For now it will work just needs memory to hold the state.

Does Nats provide the same ordering guaranty for messages

For JetStream yes, all consumption through consumers will be globally ordered. For NATS core the only ordering guarantee is per publishing connection.

[derekcollison]

Within JetStream, you have two options, lookup by sequence or by subject. Subjects can be multi-token and when filtering a stream's messages can contain wildcards.

So Its mostly in how you want to define all possible sets within a stream and make sure they represent tokens in the subjects.

We will expand the abilities for filtering messages beyond subjects in the future, but for now this can be powerful with a bit of work designing the subject space.

[Zetanova]

events.{tenant}.{arType}.{arId}.{eventType} is more or less the only subject pattern that I can think off.
where {tenant} and {eventType} are optional

The regular possibilities for the tokens are:
tenant: 2-10
arType: 5-15
arId: 10_000 - 2_000_000
eventType: 5 - 40

One AR has most of the time 3-20 messages stretch over a period of only view hours/days.
and is most of the time read over a snapshot and not replayed.

The problem/issue can simply be that the perf degenerates by the nats-index or memory pressure
and most likely it is all fine in the first year but starts in the second or 3th y.

If I have time, I will make a look into the code and/or make a benchmark.

[derekcollison]

Do you need to look up only by arid?

[derekcollison]

And if so how do you know the ID? If you assume picked by system would take it out of subject and rely on stream sequence.

[Zetanova]

this is how Event-Sourcing more or less works.
Snapshot + All AR Events from the Journal after the Snapshot.
The lookup is only over the AR-Id + SeqNr

The AR-Id's are Uuid (base36 encoded)

An other topic would be the Projections, the are most of the time queried
the tail events from the boundary domain.
I don't see an issue with projects themself with nats.

The JS stream itself would be on:
events.{boundary}.>

For the AR journal query this would be:
events.{boundary}.*.{arType}.{arId}.> + StartSeqNr

For a projection query this would be:
events.{boundary}.> + StartSeqNr

[kerimcharfi]

Is there more explanation around this? It seems very important to clarify.

How much memory overhead per stream?
Can Jetstream handle millions of individual streams (stream per individual subject)?

If millions of streams are too expensive / hard to manage:

How exactly are they indexed? (subject -> (startSeq, endSeq) will suck on sparse subjects with messages over a large interval )
How is performance for sparse subjects? (skip scan implemented?)
Can we walk backwards (looking for history) efficiently?