Hiddify 12.3.x — Bug Report Package

[navidtrc]

Hiddify 12.3.x — Bug Report Package

Prepared 2026-08-16. Findings from two servers running Hiddify Manager 12.3.2 and 12.3.3,
Ubuntu 22.04.5 LTS, x86_64. All line numbers below were verified against upstream main on the day
of writing.

---

0. Read this first — prior-art check

I searched both repos before writing. Two of the four findings are already reported, and one
already has open PRs. Filing them again would be noise. Recommended action per finding:

#	Finding	Already reported?	Recommended action
1	cmd_in_back() missing cmd argument — every background command silently dies	Yes, 3×	Do not file. Comment on #5402; ask a maintainer to merge PR #30 or PR #32
2	apply_configs can never install a newly-enabled component	No — 0 results for DO_NOT_INSTALL	File Issue A below. This is the root cause of the year-old #5104
3	WARP: inverted -f test producing mv: cannot stat	Yes, #5328	Fold into Issue B as context, credit #5328
4	WARP: default endpoint blackholed on some hosts, no fallback, no diagnostics	No specific report	File Issue B below

Existing related tickets worth linking: #5104
(WARP not installed on fresh server — same symptom, no root cause found in a year),
#5328 (the mv error),
#5508 (WARP instability, ARM64),
#5436 / #5498 (duplicates of finding 1).

The single most valuable thing you can post is Issue A — it explains #5104, which has been open
since 2025-08-19 with 8 comments and no diagnosis.

---

ISSUE A — file this one (repo: hiddify/Hiddify-Manager)

Title: apply_configs.sh can never install a component that was disabled at install time (root cause of #5104 "WARP ERROR")

Describe the bug

Enabling any optional component in the panel and pressing Apply Configs can never work if that
component was disabled during the original install. Apply runs the component's run.sh but never
its install.sh, so the component's binary is never downloaded. run.sh then invokes a binary that
does not exist and fails.

For WARP this surfaces as the widely-reported !!!!!!!!!!!!!!! WARP ERROR, which is
#5104 — open since 2025-08-19. I believe
this is its root cause. The bug is not WARP-specific; it affects every component under other/
that ships its own install.sh (ssh, telegram, dnstt, mieru, ssfaketls, warp, …).

Root cause

apply_configs.sh (line 3):

DO_NOT_INSTALL=true ./install.sh apply_configs $@

install.sh install_run() (lines 168–178):

function install_run() {
    echo "======================$1====================================={"
    if [ "$DO_NOT_INSTALL" != "true" ]; then
        runsh install.sh $@          # <-- skipped for every apply
        ...
    fi
    if [ "$DO_NOT_RUN" != "true" ]; then
        runsh run.sh $@              # <-- always runs
    fi
}

So Apply Configs is structurally incapable of provisioning a component. There is no code path that
notices "this component is newly enabled and has never been installed".

To Reproduce

1. Install Hiddify on a fresh server with WARP disabled.
2. In the panel, set WARP to all (or enable any other optional component).
3. Press Apply Configs.
4. Observe in the log:

####70####Applying...####Warp####
===run.sh other/warp
Checking WARP ...
mv: cannot stat 'wgcf-account.toml': No such file or directory
!!!!!!!!!!!!!!! WARP ERROR

5. Confirm the binary was never fetched:

$ find / -name 'wgcf*' -not -path '/proc/*'
(no output)

$ cat /opt/hiddify-manager/common/packages.db
dnstm|0.7.0
vaydns|0.2.5
xray|26.3.27          # <-- no wgcf, though it IS present in packages.lock

$ grep wgcf /opt/hiddify-manager/common/packages.lock
wgcf|2.2.30|amd64|https://github.com/ViRb3/wgcf/releases/download/v2.2.30/wgcf_2.2.30_linux_amd64|fc4430...

The package is registered and available — it is simply never downloaded.

Expected behavior

Enabling a component and applying should provision it. Either:

• install_run() runs a component's install.sh when the component is enabled but not yet
  installed (e.g. set_installed_version / packages.db has no entry for it), even under
  DO_NOT_INSTALL; or
• the panel routes "component newly enabled" through a full install rather than apply; or
• at minimum, run.sh fails loudly with "component not installed — run a full reinstall", instead
  of a bare WARP ERROR.

Workaround (confirmed on both servers)

cd /opt/hiddify-manager/other/warp/wireguard
bash install.sh     # downloads wgcf 2.2.30
bash run.sh         # registers + brings the tunnel up

Or a full reinstall, which does run every install.sh:

bash /opt/hiddify-manager/install.sh

After this, WARP came up and a subsequent Apply Configs completed with zero errors.

Environment

• Hiddify Manager 12.3.2 and 12.3.3 (both affected)
• Ubuntu 22.04.5 LTS, x86_64
• Verified against main at time of writing

---

ISSUE B — file this one (repo: hiddify/Hiddify-Manager)

Title: WARP run.sh: failures are silent and unrecoverable — swallowed errors, false-negative health check, single hardcoded endpoint with no fallback

Describe the bug

other/warp/wireguard/run.sh.j2 has several robustness problems that combine to make WARP failures
undiagnosable. On one of my servers WARP was completely non-functional while the script still
printed WARP is WORKING!, and on another the only output was WARP ERROR with no indication of
why.

B1 — Inverted file test (already reported as #5328)

Lines 8–9:

if ! [ -f "wgcf-account.toml" ]; then
  mv wgcf-account.toml wgcf-account.toml.backup   # backs up a file just proven absent

The mv is inside the branch taken when the file does not exist, so it always fails. This is the
mv: cannot stat 'wgcf-account.toml' noise in #5328. Cosmetic on its own, but it masks the real
failure and misleads everyone debugging it.

B2 — All diagnostics are discarded

./wgcf register --accept-tos -m hiddify -n $(hostname) >/dev/null 2>&1
./wgcf update >/dev/null 2>&1 || return $?
./wgcf generate >/dev/null 2>&1

When wgcf is missing or Cloudflare rejects registration, the reason is thrown away. The operator
sees only !!!!!!!!!!!!!!! WARP ERROR after three silent retries. In my case the actual error was
./wgcf: No such file or directory — trivially diagnosable, completely hidden.

Suggested fix: log stderr to $LOG_FILE rather than /dev/null, and check [ -x ./wgcf ] up
front with a clear message.

B3 — Health check gives false negatives

Line 24:

curl -s --interface warp --connect-timeout .5 http://ip-api.com?fields=...

A 0.5 s connect timeout against a cold tunnel fails routinely. On every run I observed
WARP is not working! on the first attempt and success on the retry. Worse, the check is a single
HTTP request — it passed once on a server where WARP was in fact dropping ~100% of traffic.

Suggested fix: raise the timeout to 3–5 s, and require e.g. 2 of 3 successes before declaring
the tunnel healthy.

B4 — Single hardcoded endpoint, no fallback (the actual outage)

wgcf generate writes:

Endpoint = engage.cloudflareclient.com:2408

On a host with working IPv6, that hostname resolves to an IPv6 Cloudflare endpoint. On my server
(45.x.x.x, EU VPS) that endpoint completed the WireGuard handshake but silently dropped all
payload traffic:

$ wg show warp
  latest handshake: 24 seconds ago
  transfer: 72.80 KiB received, 36.47 KiB sent    # handshake traffic only

$ ping -I warp -c 4 8.8.8.8
4 packets transmitted, 0 received, 100% packet loss

$ for i in $(seq 1 8); do curl -s --interface warp -m 8 https://v4.ident.me/; done
(8/8 empty — curl rc=28 timeout)

This is the worst failure mode: the tunnel looks healthy by every metric the script checks
(interface up, handshake recent, counters non-zero) while passing no traffic.

I scanned the WARP anycast ranges from this host:

Endpoint	Success	ICMP loss
162.159.192.1:2408	0/3	100%
162.159.193.10:2408	0/3 (no handshake)	—
162.159.195.1:2408	1/3	—
188.114.96.1:2408	10/10	0%
188.114.97.1:2408	10/10	0%
188.114.98.224:2408	9/10	0%
188.114.99.224:2408	10/10	0%

The entire 162.159.19x.x range is blackholed from this host; the 188.114.9x.x range is perfect.
run.sh has no way to discover this — it writes one endpoint and gives up.

Suggested fix: after generating the profile, probe a list of known WARP endpoints and keep the
first that passes a real connectivity test; fall back to IPv4 endpoints when the hostname resolves
to an unusable IPv6 address.

Workaround (confirmed)

Pin a working endpoint in other/warp/wireguard/run.sh.j2 — note it must be the .j2 template,
because run.sh is re-rendered from it on every apply, silently reverting any edit to run.sh:

# in generate_warp_config(), after the existing "Table = off" sed:
sed -i 's|^Endpoint = .*|Endpoint = 188.114.97.1:2408|' wgcf-profile.conf

After this: 10/10 requests succeed, 0% packet loss, and the pin survives Apply Configs.

Environment

• Hiddify Manager 12.3.2 and 12.3.3
• Ubuntu 22.04.5 LTS, x86_64
• wgcf 2.2.30, free WARP account

---

COMMENT C — post on existing issue #5104

I think I have the root cause for this.

WARP isn't installed because apply_configs.sh never runs any component's install.sh. It
calls install.sh with DO_NOT_INSTALL=true, and install_run() (install.sh:168–178) skips
runsh install.sh whenever that flag is set — it only runs run.sh.

So if WARP was disabled during the original install, other/warp/wireguard/install.sh never ran,
the wgcf binary was never downloaded, and no amount of pressing Apply Configs will ever install
it. run.sh then calls ./wgcf, which doesn't exist; the error is swallowed by
>/dev/null 2>&1, and you get !!!!!!!!!!!!!!! WARP ERROR with no explanation.

Confirm on an affected server:

$ find / -name 'wgcf*' -not -path '/proc/*'      # empty
$ cat /opt/hiddify-manager/common/packages.db    # no wgcf line
$ grep wgcf /opt/hiddify-manager/common/packages.lock   # present but never fetched

Workaround:

cd /opt/hiddify-manager/other/warp/wireguard && bash install.sh && bash run.sh

This is not WARP-specific — it affects any component under other/ that was off at install time.
Filed separately as .

---

FINDING 1 — do not file, already covered

hiddifypanel/panel/run_commander.py lines 82–83 vs 89–90 on main:

if run_in_background:
    t = threading.Thread(target=cmd_in_back, daemon=True)   # args=(base_cmd,) missing
    t.start()

def cmd_in_back(cmd):                                       # requires cmd
    p = subprocess.Popen(cmd, cwd=..., start_new_session=True)

The thread dies immediately with:

TypeError: cmd_in_back() missing 1 required positional argument: 'cmd'

Because it dies inside a thread, the traceback goes to threading.excepthook and never reaches the
panel log — so Apply Configs, Restart Services, Update, Status and Get Certificate all appear to
hang with an empty progress bar. Nothing was ever started.

Regression introduced in v12.3.0; <= v12.2.0b1 called subprocess.Popen(base_cmd, ...)
directly and worked. Still unfixed on main, beta and dev, so no released version avoids it —
downgrading does not help.

Fix (one line, already proposed in PR #30 and PR #32):

t = threading.Thread(target=cmd_in_back, args=(base_cmd,), daemon=True)

Local hotfix until a release ships it:

find /opt/hiddify-manager/.venv* -path '*hiddifypanel/panel/run_commander.py' \
  -exec sed -i 's|threading.Thread(target=cmd_in_back, daemon=True)|threading.Thread(target=cmd_in_back, args=(base_cmd,), daemon=True)|' {} + \
  && systemctl restart hiddify-panel

Rather than a new issue, the useful action is a comment on #5402 noting it is still broken in 12.3.3
and asking for PR #30/#32 to be merged.

---

Appendix — interaction between findings 1 and 2

These two compound in a way worth flagging to maintainers. On my production server, warp_mode was
set to all in the panel, but because of finding 1 the apply never executed — so the rendered
configs still routed to freedom and the setting had silently never taken effect. Fixing only
finding 1 would have made the next apply route every user into a warp interface that finding 2
had prevented from ever existing.

Correct remediation order on an affected server:

1. Patch run_commander.py, restart the panel.
2. Install and verify WARP before applying (cd other/warp/wireguard && bash install.sh && bash run.sh).
3. Verify independently — curl --interface warp https://v4.ident.me/ — not via the script's own
   real_test, whose 0.5 s timeout gives false negatives.
4. Only then run Apply Configs.