Repository navigation
Helm chart and docs omit the charges auto-advance job that quickstart runs every minute #5300
Description
Preflight Checklist
- I have searched the issue tracker for an issue that matches the one I want to file, without success.
- I am not looking for support or already pursued the available support channels without success.
OpenMeter Version
v1.0.0-beta.232. The same gap is still on main (cmd/jobs/quickstart/cronjobs.go vs deploy/charts/openmeter/templates/jobs.yaml).
Expected Behavior
When charges are enabled, non-final charges should be advanced by a periodic job, not only inside the create-charge HTTP request.
The quickstart scheduler already does this every minute, but only when the charges advancer is wired:
// cmd/jobs/quickstart/cronjobs.go
// Advance charges every minute (if charges are enabled)
if internal.App.ChargesAutoAdvancer != nil {
// ChargesAutoAdvancer.All(ctx, namespaces)
}The Helm chart should ship the same job, and deploy/charts/openmeter/values.yaml should document it next to the other billing jobs. Those comments already describe subscriptionSync as a backup reconciler for when the billing worker misses work. Charge advancement needs the same backup.
Suggested CronJob, matching the existing billing jobs:
openmeter-jobs --config /etc/openmeter/config.yaml billing advance-charges all
Schedule * * * * * (same interval as quickstart), concurrencyPolicy: Forbid.
Actual Behavior
deploy/charts/openmeter/templates/jobs.yaml only creates:
billing subscriptionsync all(hourly)billing collect all(every 5 minutes)billing advance all(invoices, every 5 minutes)
There is no CronJob for billing advance-charges. values.yaml does not mention it. A Helm install therefore has no periodic fallback.
The synchronous path is not a substitute. autoAdvanceCreatedCharges (openmeter/billing/charges/service/create.go) advances every due credit-only charge for that customer inside the create request. If one advance fails, later creates retry the whole backlog under the caller's timeout, roll back, and the backlog only grows.
AdvanceChargesEvent is handled by the billing worker (openmeter/billing/worker/worker.go), but nothing in the tree publishes that event. The worker does not scan for due charges on its own. The only automatic scanner is the quickstart cron, which the chart does not deploy.
Steps To Reproduce
- Install OpenMeter with the official Helm chart and charges/credits enabled. Do not add an extra CronJob.
- Create credit-only flat-fee charges for one customer (
POST /api/v3/.../charges,settlement_mode=credit_only). - Let one advance fail (timeout, transient DB error, or cancel the request after the charge row is committed).
- Keep creating charges for that customer.
charge_flat_feesrows staystatus=createdwithadvance_afterin the past. Each new create tries to advance all of them and times out.- Customer balance reads then spend their time in
list charges: failed to get countoncharges_search_v1s.
Manual recovery that worked: openmeter-jobs --config /etc/openmeter/config.yaml billing advance-charges customer <customer_id> --namespace default.
Additional Information
We hit this on a chart-based production install (v1.0.0-beta.232). One failed advance left a few hundred non-final flat-fee charges on a single customer. New charges and balance lookups for that customer then timed out until we ran advance-charges by hand.
Related: #4065 added the advancer, the CLI (billing advance-charges list|customer|all), and the quickstart cron. The Helm chart and its values docs were not updated to match.
Activity