Ability to Sync rETL Logs to Warehouse

[bjackowitz-caring]

File: connections/reverse-etl/manage-retl.md

We want to be able to monitor the history of our rETL syncs more proactively and natively in the tools that we use every day (i.e. - not the Segment web app). If we were able to sync the logs of our rETL runs to the warehouse, it would give us a lot more visibility into the health of those syncs/models which we could address more proactively.

[pwseg]

@bjackowitz-caring Thanks for reaching out about this. Could you tell us a bit more about your use case and what specific data you're looking to monitor? For example, are you mostly looking for which syncs succeeded, failed, or partially succeeded? or are you looking for details on specific errors/sync times?

If you can give us more info about your needs, I’ll definitely pass it along to our team.

In the meantime, while it's not currently possible to sync rETL run logs directly to your warehouse, there are some potential workarounds that might be helpful:

• Monitor emails; Segment sends out monitor emails for failed or partially successful rETL syncs. Those don't include a full log history, but they should give you a high-level overview of sync health and alert you to any issues.
• Use our API; we have a couple of endpoints you can use to get details about rETL syncs: Get Reverse ETL Sync Status, which gets details for a specific sync (and requires a sync_id), and List Reverse ETL Sync Statuses, which will give you the sync history for a specific model or subscription.

Hopefully this helps, though let me know if you have any other questions.

[bjackowitz-caring]

Looking primarily for:

• Specifics on errors - which syncs failed, and was there a specific reason?
• How many records succeeded, failed, etc for each sync
• If we can get details on the failures (i.e. - if there is a specific record or value that failed, we'd want to know that)

If we can get record level metrics that'd be awesome, but I understand that could be challenging. Something like one entry for each record we tried to sync, the outcome, and any associated error messages.