UseOptimizedProjectionRebuilds not working for customprojection since no eventypes are know. And why do we change the type of the stream for a rebuild?

[schadowfax]

Issue1: CustomProjection(s) like the documentation example (custom-aggregate) does not working with UseOptimizedProjectionRebuilds true.

the issue seems to be in the in the 'BackfillStreamTypeOperation'

_eventTypeNames = projection.AllEventTypes.Select(x => events.EventMappingFor((Type)x).EventTypeName).ToArray();

AllEventTypes are null at this point and give a exception.

Issue2: Also the logic that is executed here in the 'BackfillStreamTypeOperation' i find bit wierd.

update public.mt_streams s set type = $1 from public.mt_events e where s.id = e.stream_id and s.tenant_id = e.tenant_id and s.type is NULL and e.type = ANY($2) and s.is_archived = FALSE and e.seq_id > $3 and e.seq_id <= $4",

Why would I want to change the type of my stream? From what I understand, this is done to populate the mt_aggregate_rebuild table based on stream type information. However, this approach can introduce significant and unexpected issues.

Consider this scenario:

You have Aggregate1 and a SingleStreamViewProjection1, which subscribes to a subset of events from Aggregate1.
If you attempt to rebuild both simultaneously, concurrency issues are likely to occur, potentially leading to significant data loss.

Wouldn't it be better to use an intermediary table like mt_aggregate_rebuild to handle this behavior explicitly? This would also avoid incorrectly setting the original stream as a 'SingleStreamViewProjection1', which is inherently problematic.

[jeremydmiller]

@schadowfax I'll continue to update the docs to warn folks that the "optimized rebuilds" are a very limited feature right now and not usable in your scenario. Otherwise though, I don't think there's anything actionable here and I'm very tempted to rip out this feature altogether and only have it available through the paid model later.

[schadowfax]

@jeremydmiller for issue2 i can understand and i like the warning but can't this be resolved by having a setting that you opt in for this projection rebuild by not using the 'optimized rebuilds' for this projection? didn't look in the code for it but i think that should be quite easy depending on how its all modeld right now.

for this first issue i should also expect a easy fix if you use the IncludeType for your CustomProjection defined as is mentiond in the docs.

this optimized rebuild is quite handy since its a lot faster then a normal rebuild and on a few milion events rebuilding can now take verry long which is inherently problematic

[jeremydmiller]

@schadowfax Were you the guy that wrote this?

"
@nickchapsas so does martendb, projection rebuilding is a pain in the ass. I like the library but for large projects i wouldn't recomend. Also the patches and migrations to new versions bring some pain.

and don't forget about the documentation that is really basic and missing alot and some feature's only work in certain cases. Also i'm hearing of a paid model in the feature for martendb for some feature's so...
"

You're asking for free support on a very complicated feature that is looking like it's going to be far too difficult to really support. You cannot in good faith be griping about that. It's nice that you've contributed a little bit, but you cannot be that demanding to folks who do not work for you or your employer. Since there's no other tool on Earth that I'm aware of that really deals with projection rebuilding in any other way -- and even if there was, they'd be a paid model, feel very free to start an actual consulting engagement or partnership with JasperFx to help you build that functionality.

And to make this abundantly clear, I'm not willing to give you any additional free support here or on Discord from this moment out after that comment.

[schadowfax]

Hey @jeremydmiller , that was indeed my comment. In retrospect, I may have overstepped a bit (and updated it), and I want to sincerely apologize for that. I genuinely appreciate the time you take to respond and provide support on various topics.

You're absolutely right—we're not paying for support, and if it were up to me, I would gladly advocate (and already did once) for hiring your team to review our implementation and provide guidance. However, sadly that's not a decision I have the authority to make within my organization. I also understand that if certain features were offered under a paid model, that when I would personally love them as a developer, but I also recognize that my organization may not opt for them.

Regarding this particular issue, the challenge we're facing is that this feature seems essential for us to scale properly.(but currently looking into the blue-green projections tip from the comment also for this) Based on the documentation, we understood that it would be the standard of V8 so that's why we are looking into it. Given that it was documented as such, I had no way of knowing that it was initially developed for a specific client. Now, I'm encountering a few scenarios where certain approaches don't seem to work as expected. Specifically, use case 1 is problematic since i can't opt out just for my custom projection unless i register multiple store's perhaps.

Additionally, another team has experienced inconsistencies when rebuilding projections—whether using a single stream rebuild, a normal rebuild, or an optimized rebuild. The result has variations, though small, have been frustrating for certain use cases.

One major issue we're currently facing is a rebuild process that clears the table but never reinserts any data. This eventually leads to crashes when applying events because there's no valid initial object to work with—it just defaults to a new object. It appears to be related to the upsert mechanism on our snapshots, as other rebuilds work fine.

I suspect the issue lies with the current_version being greater than the revision (since it seems to be always 1 but the stream is mutch further ofcourse on a rebuild). Here's the relevant query snippet we're looking at from the upsert function for that table:

SELECT version INTO current_version
FROM public.mt_streams
WHERE id = docId
AND public.mt_streams.tenant_id = tenantid;

IF revision = 0 THEN
IF current_version IS NOT NULL THEN
revision = current_version;
ELSE
revision = 1;
END IF;
ELSE
IF current_version IS NOT NULL THEN
IF current_version > revision THEN
RETURN 0;
END IF;
END IF;
END IF;

We've currently stabilized our environment, and when I find the time, I'll check if this issue still persists. If it does, I'll dive deeper into it and, if possible, create a new ticket or pull request based on my understanding of the framework—just as I attempted before.

Once again, I sincerely apologize for how my previous message came across. It was never my intention to be disrespectful, and I completely understand your reaction and the implications.

[jeremydmiller]

Reopening this to make sure something gets done, even if it's just temporarily documentation. The optimized rebuild functionality is currently not providing much value, but does have some value.

@schadowfax I appreciate that, and I was cooled off this morning even before seeing your message. We'll do something to improve the situation, and I do like your suggestions.