Repository navigation
fix(tools): eliminate UBSan -fsanitize=function violations in vidinput vtbl - #180
Merged
Merged
Conversation
lusoris
enabled auto-merge (squash)
May 29, 2026 10:25
lusoris
disabled auto-merge
May 29, 2026 11:43
lusoris
marked this pull request as draft
May 29, 2026 11:43
lusoris
force-pushed
the
fix/ubsan-vidinput-vtbl-type-mismatch
branch
from
May 29, 2026 12:12
31f070a to
f7e9ce3
Compare
Contributor
Author
|
Contaminated (33 files) — needs reconstruction, skipping rebase per session policy |
lusoris
marked this pull request as ready for review
May 31, 2026 13:38
Contributor
Author
|
Superseded by master after merge marathon 2026-05-31. |
…t vtbl yuv_input.c and y4m_input.c registered functions using concrete `yuv_input *` / `y4m_input *` parameter types in YUV_INPUT_VTBL / Y4M_INPUT_VTBL, while the vidinput.h typedefs declare the context parameter as `void *`. UBSan -fsanitize=function detects every vtbl dispatch as an indirect call through a mismatched function pointer type. Fix: add static vtbl-compatible wrapper functions that accept the `void *` context and delegate to the typed concrete implementations. Remove all C-style casts from the VTBL initialisers. Pre-existing issue (inherited from upstream Netflix/vmaf Daala tooling); not introduced by any recent fork PR. no rebase impact: tool-only files, no ABI or public-header change Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lusoris
force-pushed
the
fix/ubsan-vidinput-vtbl-type-mismatch
branch
from
June 3, 2026 13:08
f7e9ce3 to
3af229c
Compare
lusoris
marked this pull request as ready for review
June 3, 2026 13:09
There was a problem hiding this comment.
Pull request overview
Eliminates UBSan -fsanitize=function undefined-behavior reports caused by calling through video_input_vtbl function pointers whose concrete implementations used incompatible parameter types (yuv_input * / y4m_input * vs the void * typedefs in vidinput.h). The fix introduces vtbl-compatible wrapper functions in the YUV and Y4M input implementations and removes the type-punning casts from the vtbl initializers.
Changes:
- Add
staticvtbl wrapper functions incore/tools/yuv_input.cthat match thevidinput.htypedefs and forward to typed implementations. - Add
staticvtbl wrapper functions incore/tools/y4m_input.cwith the same approach. - Add a changelog fragment documenting the sanitizer fix and the fact that golden scores are unchanged.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| core/tools/yuv_input.c | Adds vtbl-compatible wrappers and updates YUV_INPUT_VTBL to use them (removes function-pointer casts). |
| core/tools/y4m_input.c | Adds vtbl-compatible wrappers and updates Y4M_INPUT_VTBL to use them (removes function-pointer casts). |
| changelog.d/fixed/ubsan-vidinput-vtbl-function-ptr-mismatch.md | Documents the UBSan function-pointer-type mismatch fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lusoris
added a commit
that referenced
this pull request
Jun 12, 2026
…CI) (#868) * fix(codeql): resolve HIGH-severity security-cpp-high alerts (23 sites) - cpp/integer-multiplication-cast-to-long (11): pre-cast one operand to size_t / double / ptrdiff_t before int*int multiplications in cambi.c, float_vif.c (log message), iqa/convolve.c (img_offset), moment.c, psnr.c, and vif_tools.c (four memcpy size expressions). Add stddef.h to convolve.c for ptrdiff_t. - cpp/incomplete-parity-check (3): change `% 2 == 1` to `% 2 != 0` in vif_tools.c (assert), svm.cpp (powi loop), pdjson.c (JSON object key/value alternation). The == 1 form is wrong for negative operands; != 0 is always correct. - cpp/wrong-type-format-argument (2): fix float_vif.c error log that printed size_t fields scaled_w/scaled_h with %d; change to %zu. - cpp/world-writable-file-creation (1): in vmaf.cpp replace bare fopen("wb") with open(O_WRONLY|O_CREAT|O_TRUNC, 0644)+fdopen() on POSIX so the created file is never world-writable independent of the caller's umask. Add <fcntl.h>. - cpp/path-injection (4): in test_output.c resolve the mkstemp-created path through realpath() immediately after creation, breaking the taint chain from getenv("TMPDIR") to the vmaf_write_output call site. - cpp/toctou-race-condition (2): skipped — both sites are in test cleanup (RMDIR after stat assertion). The stat result drives a test assertion, not a security-sensitive access decision; no atomic replacement of open() is applicable to rmdir. Reported as skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(codeql): resolve security-python-and-ci CodeQL alerts Fixes 18 open CodeQL alerts across the Python and CI categories: - yaml.github-actions.security.run-shell-injection (#661): move github.event_name, github.base_ref, and github.event.before from inline ${{...}} interpolation to env: vars in the SYCL clang-tidy detect step of lint-and-format.yml. - python.lang.security.use-defused-xml-parse (#216, #217): replace xml.etree.ElementTree with defusedxml.ElementTree in feature_extractor.py and quality_runner.py; add defusedxml>=0.7.1 to python/pyproject.toml and python/requirements.txt. - py/undefined-export (#352, #353, #354, #616): restructure aiutils/__init__.py to do a conditional eager import of the parquet helpers so the names are defined when pyarrow is present, and only include them in __all__ when the import succeeded. - py/stack-trace-exposure (#178, #179, #585): log exception detail server-side and return a generic message to the HTTP client in http_transport.py _handle_score (invalid JSON, bad params, scorer error branches). - python.lang.security.audit.dangerous-subprocess-use-tainted-env-args (#227, #372): add shlex.quote() around user-supplied path arguments passed into shell strings in extract_ugc_features.py and test_bbb_e2e_v5_bug_cluster.py. - py/file-not-closed (#677, #678): replace bare open() calls with context managers in test_coverage_round3.py. - py/redundant-comparison (#427, #431): remove redundant assert not (x != y) lines that duplicate the preceding assert x == y. - py/equals-hash-mismatch (#182): convert RdPoint to frozen=True dataclass so __eq__ and __hash__ are generated consistently. - py/inheritance/signature-mismatch (#197): add result_dict=None default to EnsembleVmafQualityRunner._populate_result_dict so the signature is compatible with the base class. - py/multiple-definition (#201): drop redundant assignment to feature_found in feature_extractor.py wildcard discovery path. - py/str-format/surplus-named-argument (#204): remove unused dataset= kwarg from the format() call in routine.py. Skipped: python.lang.security.audit.insecure-file-permissions (#373) — the Unix socket at 0o660 is intentional (Go sidecar node must write to it and runs as the same UNIX group); tightening to 0o644 would break the IPC channel. Skipped: py/path-injection (#180, #181) — _validate_path() already resolves the path and checks it against an allowlist before any file operation; the data flow is secure and the CodeQL dataflow trace is a false positive on this allowlisted pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Lusoris <lusoris@pm.me> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
yuv_input.candy4m_input.cregistered vtbl functions using concreteyuv_input */y4m_input *parameter types, whilevidinput.htypedefs declare the context parameter asvoid *. UBSan-fsanitize=functiondetects every vtbl dispatch as an indirect call through a mismatched function pointer type — 4 runtime errors per Netflix golden test run.void *and delegate to the typed concrete implementations. Removes all C-style casts from the VTBL initialisers.Sanitizer run results
Before fix (ASan+UBSan, HEAD of master before this PR):
4 UBSan reports per golden run, across all 3 Netflix golden pairs (vidinput.c lines 36, 63, 68, 78).
After fix: 0 sanitizer findings. All 3 golden pairs produce correct scores:
TSan: 0 data races, 0 deadlocks (all 3 golden runs, separate TSan build).
ASan leaks: 0 (confirms PR #94 pinned-host leak fix is effective).
MSan: Not feasible on this host — requires fully-instrumented libc; uninstrumented system libraries produce false positives that are indistinguishable from real findings.
Reproducer
Checklist
changelog.d/fixed/ubsan-vidinput-vtbl-function-ptr-mismatch.md🤖 Generated with Claude Code