Fix: a scenario's before_scenario hook failure wrongly fails its siblings under --parallel-scheme feature

[taltal-beep]

Summary

Under --parallel-scheme feature with more than one worker process, a single scenario's before_scenario hook failure causes every other scenario in the same feature to be misreported as failed too — including scenarios that ran and passed cleanly. --parallel-scheme scenario is unaffected.

Found while running a suite with a stochastic failure-injection hook (before_scenario occasionally raises to simulate flaky setup) — a BehaveX stage consistently reported "0 passed, N failed" for a feature where, per the actual random draws, only a fraction of scenarios should have failed. The exact same hook, same repo, run via native single-process behave, correctly reported a normal mix of passed/hook_error.

Root cause

runner.hook_failures is a blunt counter behave's ModelRunner.run_hook() increments for any hook exception, regardless of hook type (before_all, before_scenario, before_step, tag hooks, ...) — see behave's runner.py. _calculate_execution_code_from_runner() treats hook_failures > 0 as execution_code = 2 ("crashed"). execute_tests() then discards the real, already-correctly-computed per-scenario JSON and replaces it with a synthetic skeleton marking every scenario in that feature as 'failed'.

Under --parallel-scheme feature, one worker runs every scenario of a feature in a single behave invocation, so hook_failures accumulates across all of them. A single scenario's before_scenario failure — which behave itself already isolates correctly to just that one scenario (hook_failed on the scenario, execution continues normally) — is enough to poison the whole feature's reported results.

--parallel-scheme scenario never hit this because each scenario gets its own isolated runner.run() call — no siblings to poison.

Minimal repro

Feature: Sample hook failure isolation

  Scenario: First scenario passes cleanly
    Given a trivial step

  Scenario: Second scenario fails its before_scenario hook
    Given a trivial step

  Scenario: Third scenario should also pass cleanly
    Given a trivial step

def before_scenario(context, scenario):
    if scenario.name == "Second scenario fails its before_scenario hook":
        raise AssertionError("Intentional before_scenario failure for repro")

$ behavex --parallel-processes 2 --parallel-scheme feature features
...
Failing scenarios:
  features/sample.feature:3  First scenario passes cleanly
  features/sample.feature:6  Second scenario fails its before_scenario hook
  features/sample.feature:9  Third scenario should also pass cleanly

0 features passed, 1 failed, 0 skipped
0 scenarios passed, 3 failed, 0 skipped
0 steps passed, 0 failed, 0 skipped, 3 untested
Exit code: 1

Expected (and what --parallel-scheme scenario / native behave correctly produce): 2 passed, 1 failed.

Fix

Two changes, both needed together (the second was masked by the first and only surfaced once real per-scenario results were allowed through):

1. execute_tests(): only fall back to the synthetic "everything failed" skeleton when there really are no real results to report (e.g. the runner crashed/aborted before producing any feature data). When behave completed normally and handed back real per-scenario results despite hook_failures > 0, those real results are used instead. execution_code == 2's existing severity signal is preserved for hook types that aren't scenario-scoped (before_all, after_all, before_feature, after_feature, tag hooks) — those don't show up in any individual scenario's status, so they still need this signal to correctly drive the final exit code. (Confirmed via the existing @CRASHING_BEHAVE_HOOK outline in tests/features/crashing_tests.feature, which covers exactly this — see Verification.)

2. Scenario-status classification (used to build the final totals and failing_non_muted_tests): didn't recognize behave's own hook_error/cleanup_error status values (see behave's model_type.py Status enum and is_hook_error()), silently bucketing them as "skipped" and never flagging the run as failed. Fixed to count them as failures, matching behave's own semantics.

Test

Added tests/features/partial_hook_failures.feature + tests/features/partial_hook_failures/ (feature, environment.py, steps) — 3 scenarios in one feature where only the middle one's before_scenario hook fails, run with --parallel-processes 2 --parallel-scheme feature, asserting 2 scenarios passed, 1 failed, 0 skipped and exit code 1 (fails on master, passes with this fix).

Verification

• Minimal repro above, both --parallel-scheme feature and --parallel-scheme scenario, now produce identical, correct 2 passed, 1 failed / exit code 1.
• The full existing tests/features/crashing_tests.feature @CRASHING_BEHAVE_HOOK scenario outline (20 examples: every hook type × 1/2 parallel processes) still passes unchanged — confirms genuinely fatal hook failures (before_all, after_all, etc.) still correctly report exit code 1.
• Full tests/features/ suite (334 scenarios) diffed scenario-by-scenario against master: zero regressions — the only status change anywhere in the suite is the new regression test itself flipping from incorrectly-erroring to correctly-passing.