Repository navigation
chore(ai): parquet schema v2 — canonical column order + zstd-3 + schema metadata - #427
Merged
Merged
Conversation
lusoris
marked this pull request as ready for review
May 31, 2026 03:33
lusoris
enabled auto-merge (squash)
May 31, 2026 03:33
lusoris
force-pushed
the
chore/parquet-schema-v2
branch
from
May 31, 2026 07:31
bb5318e to
49c14b1
Compare
lusoris
marked this pull request as draft
May 31, 2026 09:02
auto-merge was automatically disabled
May 31, 2026 09:02
Pull request was converted to draft
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| __all__ = [ | ||
| "add_batch_manifest_arguments", | ||
| "apply_standard_column_order", |
| "build_run_provenance", | ||
| "collect_cli_argv", | ||
| "describe_path", | ||
| "detect_schema_version", |
| "iter_jsonl", | ||
| "make_argument_parser", | ||
| "now_iso_8601", | ||
| "read_parquet_with_schema", |
lusoris
marked this pull request as ready for review
May 31, 2026 12:09
lusoris
force-pushed
the
chore/parquet-schema-v2
branch
from
May 31, 2026 12:12
49c14b1 to
67badc6
Compare
lusoris
added a commit
that referenced
this pull request
Jun 12, 2026
…CI) (#868) * fix(codeql): resolve HIGH-severity security-cpp-high alerts (23 sites) - cpp/integer-multiplication-cast-to-long (11): pre-cast one operand to size_t / double / ptrdiff_t before int*int multiplications in cambi.c, float_vif.c (log message), iqa/convolve.c (img_offset), moment.c, psnr.c, and vif_tools.c (four memcpy size expressions). Add stddef.h to convolve.c for ptrdiff_t. - cpp/incomplete-parity-check (3): change `% 2 == 1` to `% 2 != 0` in vif_tools.c (assert), svm.cpp (powi loop), pdjson.c (JSON object key/value alternation). The == 1 form is wrong for negative operands; != 0 is always correct. - cpp/wrong-type-format-argument (2): fix float_vif.c error log that printed size_t fields scaled_w/scaled_h with %d; change to %zu. - cpp/world-writable-file-creation (1): in vmaf.cpp replace bare fopen("wb") with open(O_WRONLY|O_CREAT|O_TRUNC, 0644)+fdopen() on POSIX so the created file is never world-writable independent of the caller's umask. Add <fcntl.h>. - cpp/path-injection (4): in test_output.c resolve the mkstemp-created path through realpath() immediately after creation, breaking the taint chain from getenv("TMPDIR") to the vmaf_write_output call site. - cpp/toctou-race-condition (2): skipped — both sites are in test cleanup (RMDIR after stat assertion). The stat result drives a test assertion, not a security-sensitive access decision; no atomic replacement of open() is applicable to rmdir. Reported as skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(codeql): resolve security-python-and-ci CodeQL alerts Fixes 18 open CodeQL alerts across the Python and CI categories: - yaml.github-actions.security.run-shell-injection (#661): move github.event_name, github.base_ref, and github.event.before from inline ${{...}} interpolation to env: vars in the SYCL clang-tidy detect step of lint-and-format.yml. - python.lang.security.use-defused-xml-parse (#216, #217): replace xml.etree.ElementTree with defusedxml.ElementTree in feature_extractor.py and quality_runner.py; add defusedxml>=0.7.1 to python/pyproject.toml and python/requirements.txt. - py/undefined-export (#352, #353, #354, #616): restructure aiutils/__init__.py to do a conditional eager import of the parquet helpers so the names are defined when pyarrow is present, and only include them in __all__ when the import succeeded. - py/stack-trace-exposure (#178, #179, #585): log exception detail server-side and return a generic message to the HTTP client in http_transport.py _handle_score (invalid JSON, bad params, scorer error branches). - python.lang.security.audit.dangerous-subprocess-use-tainted-env-args (#227, #372): add shlex.quote() around user-supplied path arguments passed into shell strings in extract_ugc_features.py and test_bbb_e2e_v5_bug_cluster.py. - py/file-not-closed (#677, #678): replace bare open() calls with context managers in test_coverage_round3.py. - py/redundant-comparison (#427, #431): remove redundant assert not (x != y) lines that duplicate the preceding assert x == y. - py/equals-hash-mismatch (#182): convert RdPoint to frozen=True dataclass so __eq__ and __hash__ are generated consistently. - py/inheritance/signature-mismatch (#197): add result_dict=None default to EnsembleVmafQualityRunner._populate_result_dict so the signature is compatible with the base class. - py/multiple-definition (#201): drop redundant assignment to feature_found in feature_extractor.py wildcard discovery path. - py/str-format/surplus-named-argument (#204): remove unused dataset= kwarg from the format() call in routine.py. Skipped: python.lang.security.audit.insecure-file-permissions (#373) — the Unix socket at 0o660 is intentional (Go sidecar node must write to it and runs as the same UNIX group); tightening to 0o644 would break the IPC channel. Skipped: py/path-injection (#180, #181) — _validate_path() already resolves the path and checks it against an allowlist before any file operation; the data flow is secure and the CodeQL dataflow trace is a false positive on this allowlisted pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Lusoris <lusoris@pm.me> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of parquet schema: canonical column ordering enforced, zstd compression level 3 (was default 1), schema version metadata embedded for forward compatibility. 15 new parquet schema tests.
no state delta: per CLAUDE.md §12 r8; no bug tracker entry.ADR-0108 deliverables checklist
🤖 Generated with Claude Code