fix(settler): send admin key and widen unsettled-epoch lookback (#7396)

[lequangsang01]

Closes #7396

RTC wallet: RTCfe13452d122263caf633ab1876bd9631133b68b

Changes

• Added RC_ADMIN_KEY header (X-Admin-Key) to settle_epoch_via_api() in node/auto_epoch_settler.py — the /rewards/settle endpoint requires admin authentication but the auto-settler was sending unauthenticated requests, causing silent 401 failures
• Added explicit 401 logging with actionable guidance
• Widened unsettled-epoch lookback from 10 to 500 (configurable via RUSTCHAIN_UNSETTLED_LOOKBACK env var) to prevent epochs from being silently missed during extended settler downtime

Root cause

The auto epoch settler daemon was calling /rewards/settle without the required X-Admin-Key header, resulting in HTTP 401 rejections. Additionally, the lookback window of 10 epochs was too narrow, causing any epoch older than 10 epochs behind current to never be retried.

Testing

• Verified admin key is forwarded from RC_ADMIN_KEY env var
• Verified 401 responses are logged with clear message
• Verified lookback window is configurable

[FakerHideInBush]

Two concerns before merging:

1. A default UNSETTLED_LOOKBACK of 500 epochs means get_unsettled_epochs() now runs up to 1000 database queries every CHECK_INTERVAL (300s) — a 50× increase from the prior 10-epoch window

The function loops over range(max(0, current_epoch - 500), current_epoch) and executes 2 SELECT COUNT(*) queries per epoch:

for epoch in range(max(0, current_epoch - UNSETTLED_LOOKBACK), current_epoch):
    headers = db.execute(...)   # query 1
    already_settled = db.execute(...)  # query 2

With UNSETTLED_LOOKBACK=500, this is up to 1000 DB queries every 5 minutes. On a node with a large headers or epochs table, each query may require a full sequential scan if the slot-range index is not selective. This can cause the settler to consume significant CPU and I/O on every check cycle — especially if epochs are already settled and the 500-epoch window consistently yields zero unsettled results (all 1000 queries return 0 rows but still run).

Recommendations:

• Add a second env var RUSTCHAIN_UNSETTLED_LOOKBACK_MAX capped at a safe value (e.g. 50) with a warning if the configured value is higher.
• Or replace the per-epoch loop with a single SQL query:

SELECT DISTINCT (slot / ?) AS epoch
FROM headers
WHERE slot >= ? AND epoch NOT IN (SELECT epoch FROM settled_epochs)

2. The 401 error log message conflates 'no admin key configured' with 'wrong admin key provided' — the user gets misleading guidance

Current message:

logger.warning("Settlement for epoch %d rejected: admin key required (set RC_ADMIN_KEY)", epoch)

This message fires whenever the API returns 401. But if RC_ADMIN_KEY is already set (non-empty admin_key), the 401 means the key is wrong (expired, revoked, or malformed), not missing. Telling the operator to 'set RC_ADMIN_KEY' when it IS set (but incorrect) will cause confusion. Distinguish the two cases:

if admin_key:
    logger.warning("Settlement rejected for epoch %d: admin key was provided but rejected (401) — verify RC_ADMIN_KEY is correct", epoch)
else:
    logger.warning("Settlement rejected for epoch %d: admin key required — set RC_ADMIN_KEY env var", epoch)

[jaxint]

Review Summary

This PR fixes epoch settler configuration and authentication.

Changes:

1. Admin Key Support: Adds X-Admin-Key header for settlement API authentication
2. Lookback Config: New UNSETTLED_LOOKBACK env var (default 500) replaces hardcoded 10
3. Error Handling: Clear warning for 401 unauthorized responses

Quality:

• Configurable via environment variables
• Proper error messages
• Clean implementation

✅ APPROVED

[github-actions]

RTC Reward

This merged PR earned 5 RTC — sent to RTCfe13452d122263caf633ab1876bd9631133b68b.

RustChain Bounty Program