Skip to content

bug: five settings are accepted and persisted but have no runtime effect #10840

Open

Description

@S-a-r-i-u-s

Describe the bug

Five settings can be set, are persisted, and read back correctly via the API — but have no effect at runtime. They fail silently, which makes them worse than a missing feature: an operator configures them, verifies they are stored, and reasonably concludes the system is protected.

Setting Level Observed effect
Error handler Workspace none — never fires
on_failure Schedule none — never fires
concurrent_limit Script and Flow none — no limiting observed
cache_ttl Flow step none (on a script it works correctly)
delete_after_secs Flow none — nothing is deleted

Filing these together because the pattern is the actual finding — there is no obvious rule that predicts which fields are wired up. Happy to split into separate issues if that is easier to triage.

To reproduce

Each was tested with a control run — once with the setting, once without — since testing only the configured case measures the environment rather than the setting.

delete_after_secs is the clearest to reproduce: set it to 30 on a flow, deploy, confirm via API that delete_after_secs = 30 is returned. Run the flow. After 160 seconds: flow job present, result present, step job present, flow status present. Nothing deleted, nothing redacted.

concurrent_limit: set on both a script and a flow, then start more runs than the limit allows. All run concurrently.

cache_ttl: has no effect when set on a flow step; the same value in a script's script.yaml correctly freezes results.

Error handler / on_failure: neither the workspace error handler nor a schedule's on_failure runs when a job fails. Note that the error_handler_path on an HTTP trigger does fire correctly with the same bare path format (f/foo/handler), so path formatting is not the cause here.

Expected behavior

Either the settings take effect, or they are rejected at write time.

Screenshots

No response

Browser information

Not applicable — reproduced via API.

Application version

CE v1.792.2 (self-hosted, Docker Compose)

Additional Context

For contrast, these were verified to work correctly in the same test pass: timeout, retry, suspend, mock, continue_on_error and skip_failures on a step, same_worker on a flow, and cache_ttl on a script.

There is no discernible pattern — it is not "all flow-level fields" or "all workspace settings". same_worker is flow-level and works; concurrent_limit is script-level and does not.

Possibly related to #7311 and #5453 for the error-handler part.

Activity

  1. S-a-r-i-u-s commented on Aug 26, 2026

    @S-a-r-i-u-s
    Author

    Re-verified on CE v1.796.0-12-gffdf17ef8d (image ghcr.io/windmill-labs/windmill:main, digest sha256:f8513becee64ac47b0099a84adde4bb90962f396768070671b0ab9790e343f87), on a fresh stack from the repo's own docker-compose.yml. Two of the five confirmed still present, this time with a control run.

    concurrent_limit — still no effect, and not because of a missing time window

    The obvious objection to the original report was that concurrency_time_window_s wasn't set. It was tested both ways, plus a control with no limit at all. Script sleeps 5s; three runs fired together; three default workers available, so real parallelism is possible.

    Case Stored (read back via API) Spread, first → last start Result
    A: concurrent_limit=1, no window concurrent_limit=1, window=None 0.1 s parallel
    B: concurrent_limit=1, concurrency_time_window_s=60 concurrent_limit=1, window=60 0.0 s parallel
    C: control, no limit set — 0.0 s parallel

    All three durations were ~5040 ms and the start timestamps land within 100 ms of each other in every case. A and B are indistinguishable from C.

    delete_after_secs — still no effect

    Set to 30 on a flow, confirmed via GET /flows/get/{path} that delete_after_secs=30 came back. Ran the flow, waited 170 s, then GET /jobs_u/get/{id}:

    HTTP 200 — type=CompletedJob, result={'ok': True}
    

    Nothing deleted, nothing redacted.

    Method note

    Worth flagging for anyone reproducing this: a newly created script is not immediately runnable. Windmill starts an automatic dependencies job on create, and runs fired before it finishes never execute. In a first attempt the only "completed" job was the lock job (Successful lock file generation, 48 ms), which is easy to mistake for the script having run. Waiting for a successful trial run before measuring avoids this.

    Happy to split this issue per setting if that's easier to triage, and to test the remaining three (cache_ttl on a flow step, workspace error handler, schedule on_failure) the same way.

  2. rubenfiszel commented on Aug 26, 2026

    @rubenfiszel
    Contributor

    Thanks for the careful writeup, and in particular for the control runs.

    Four of the five are behaving as designed: they are Enterprise Edition features, and CE builds compile the enforcement out entirely. The fifth, cache_ttl on a flow step, does not reproduce for me.

    I checked against a CE build of main (no enterprise cargo feature, same shape as the published CE image).

    The four EE ones

    Setting Where it is gated
    Workspace error handler apply_completed_job_error_handlers, behind #[cfg(all(feature = "enterprise", feature = "private"))] — backend/windmill-queue/src/jobs.rs:1089
    Schedule on_failure (also on_success / on_recovery) apply_schedule_handlers, same gate — backend/windmill-queue/src/jobs.rs:1424
    concurrent_limit / concurrency_time_window_s #[cfg(not(feature = "enterprise"))] branch — backend/windmill-queue/src/jobs.rs:4247
    delete_after_secs > 0 both halves are EE: the flow-completion block that schedules the deletion (backend/windmill-worker/src/worker_flow.rs:1833) and the sweeper cleanup_scheduled_job_deletions (backend/src/monitor.rs:1863)

    Details from the repro:

    • concurrent_limit: on CE the server does log this, at ERROR level, once per pulled job:
      Concurrent limits are an EE feature only, ignoring constraints
      
      So it is not fully silent, but nothing surfaces through the API or UI, which is what you measured. Documented as Cloud/EE: https://www.windmill.dev/docs/core_concepts/concurrency_limits
    • delete_after_secs: on CE nothing is even queued for deletion. After a flow run with delete_after_secs: 15, job_delete_schedule stayed empty and the result was intact well past the deadline. Worth knowing: delete_after_secs: 0 (immediate deletion, the old delete_after_use behaviour) is inline in the flow-completion path and does work on CE.
    • Schedule on_failure: confirmed with a 0 * * * * * schedule on a throwing script. The scheduled job failed, no handler job was pushed. Documented as EE: https://www.windmill.dev/docs/core_concepts/error_handling
    • Workspace error handler: same, no handler job pushed after a failing run. Workspace settings does show an "Workspace error/success handler is an EE feature" banner in the UI, but POST /workspaces/edit_error_handler returns 200 on CE and persists the value.
    • Your note that error_handler_path on an HTTP trigger fires correctly is right and consistent: trigger-level error handlers live in windmill-trigger and have no EE gate. Different code path from the workspace and schedule handlers.

    cache_ttl on a flow step: could not reproduce

    On the same CE build, a step with cache_ttl: 300 returned an identical Math.random() value across three runs, in all three shapes:

    • inline (rawscript) step with cache_ttl set on the module
    • step referencing a workspace script that has cache_ttl set on the script
    • step referencing a workspace script with cache_ttl overridden on the module

    The cache key is (job kind, script hash or path+code, hashed args). So a step whose resolved inputs differ between runs (anything derived from flow_input, a previous step's result, or a timestamp) misses every time even though the setting is live, which is the most likely explanation. If you still see it with args that are identical run to run, please open a separate issue with the flow value and I will dig in, that would be a real bug.

    On the actual finding

    You are right that there is no rule you could have inferred, and that is the part worth fixing. The UI gates all four (eeOnly toggles, disabled inputs, the EE banner in workspace settings), but the API accepts and persists them with a 200 on CE, so configuring through the API or the CLI gives no signal whatsoever. I will look at surfacing this at write time on CE rather than leaving it to be discovered by measurement.

    Filing it as one pattern rather than five reports was the right call, and the control runs are what made it quick to triage. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions