Skip to content

Helm chart and docs omit the charges auto-advance job that quickstart runs every minute #5300

Open

Description

@wangzongming

Preflight Checklist

  • I have searched the issue tracker for an issue that matches the one I want to file, without success.
  • I am not looking for support or already pursued the available support channels without success.

OpenMeter Version

v1.0.0-beta.232. The same gap is still on main (cmd/jobs/quickstart/cronjobs.go vs deploy/charts/openmeter/templates/jobs.yaml).

Expected Behavior

When charges are enabled, non-final charges should be advanced by a periodic job, not only inside the create-charge HTTP request.

The quickstart scheduler already does this every minute, but only when the charges advancer is wired:

// cmd/jobs/quickstart/cronjobs.go
// Advance charges every minute (if charges are enabled)
if internal.App.ChargesAutoAdvancer != nil {
    // ChargesAutoAdvancer.All(ctx, namespaces)
}

The Helm chart should ship the same job, and deploy/charts/openmeter/values.yaml should document it next to the other billing jobs. Those comments already describe subscriptionSync as a backup reconciler for when the billing worker misses work. Charge advancement needs the same backup.

Suggested CronJob, matching the existing billing jobs:

openmeter-jobs --config /etc/openmeter/config.yaml billing advance-charges all

Schedule * * * * * (same interval as quickstart), concurrencyPolicy: Forbid.

Actual Behavior

deploy/charts/openmeter/templates/jobs.yaml only creates:

  • billing subscriptionsync all (hourly)
  • billing collect all (every 5 minutes)
  • billing advance all (invoices, every 5 minutes)

There is no CronJob for billing advance-charges. values.yaml does not mention it. A Helm install therefore has no periodic fallback.

The synchronous path is not a substitute. autoAdvanceCreatedCharges (openmeter/billing/charges/service/create.go) advances every due credit-only charge for that customer inside the create request. If one advance fails, later creates retry the whole backlog under the caller's timeout, roll back, and the backlog only grows.

AdvanceChargesEvent is handled by the billing worker (openmeter/billing/worker/worker.go), but nothing in the tree publishes that event. The worker does not scan for due charges on its own. The only automatic scanner is the quickstart cron, which the chart does not deploy.

Steps To Reproduce

  1. Install OpenMeter with the official Helm chart and charges/credits enabled. Do not add an extra CronJob.
  2. Create credit-only flat-fee charges for one customer (POST /api/v3/.../charges, settlement_mode=credit_only).
  3. Let one advance fail (timeout, transient DB error, or cancel the request after the charge row is committed).
  4. Keep creating charges for that customer.
  5. charge_flat_fees rows stay status=created with advance_after in the past. Each new create tries to advance all of them and times out.
  6. Customer balance reads then spend their time in list charges: failed to get count on charges_search_v1s.

Manual recovery that worked: openmeter-jobs --config /etc/openmeter/config.yaml billing advance-charges customer <customer_id> --namespace default.

Additional Information

We hit this on a chart-based production install (v1.0.0-beta.232). One failed advance left a few hundred non-final flat-fee charges on a single customer. New charges and balance lookups for that customer then timed out until we ran advance-charges by hand.

Related: #4065 added the advancer, the CLI (billing advance-charges list|customer|all), and the quickstart cron. The Helm chart and its values docs were not updated to match.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions