feat: Introduce setting for max record age on targets

[dluo-sig]

Feature scope

Taps (catalog, state, stream maps, tests, etc.)

Description

https://github.com/meltano/sdk/blob/main/singer_sdk/target_base.py#L53

_MAX_RECORD_AGE_IN_MINUTES is hard-coded and does not allow for configuration. I'm not sure why this would be enforced, as I would normally expect that the records that I'm pulling would be consistent as of the time of the original query. Maybe allow 0 or -1 to indicate to never time out. That would let things operate on configured batch size. This would also allow for some performance improvements as we have noticed that upsert performance is better than linear with the number of records.

[dluo-sig]

It seems like it was changed from 30 to 5 here. #1064

Not sure if I'm understanding correctly, but based on this comment, it seems like the batch flush operation is a performance penalty as opposed to finishing up the batch? I do see this happen every 5 minutes and it takes around 40 seconds before I see the next log message on my target starting to process the batch.

[ashish-atidiv]

I too believe _MAX_RECORD_AGE_IN_MINUTES should be configurable.

[edgarrmondragon]

I've added Accepting Pull Requests , in case anyone's motivated to tackle it themselves 🙂

[dluo-sig]

It seems like it was changed from 30 to 5 here. #1064

Not sure if I'm understanding correctly, but based on this comment, it seems like the batch flush operation is a performance penalty as opposed to finishing up the batch? I do see this happen every 5 minutes and it takes around 40 seconds before I see the next log message on my target starting to process the batch.

[ashish-atidiv]

I too believe _MAX_RECORD_AGE_IN_MINUTES should be configurable.

[edgarrmondragon]

I've added Accepting Pull Requests , in case anyone's motivated to tackle it themselves 🙂