Skip to content

fix: harden sync workflow and snapshot error handling - #4

Merged
dongjiang1989 merged 2 commits into
masterfrom
fix-pre-dryrun-fail
May 29, 2026
Merged

fix: harden sync workflow and snapshot error handling#4
dongjiang1989 merged 2 commits into
masterfrom
fix-pre-dryrun-fail

Conversation

@dongjiang1989

@dongjiang1989 dongjiang1989 commented May 29, 2026

Copy link
Copy Markdown
Member

Closes: #3

sync.yml:

sync_engine.py:

  • Wrap HF snapshot fetch in try/except (was unprotected, unlike MS)
  • Add explicit guard: HF_TO_MS fails clearly when HF snapshot unavailable
  • Add explicit guard: BIDIRECTIONAL fails clearly when both snapshots unavailable
  • Fix _update_state: handle None hf_snapshot (was causing AttributeError)
  • Change exit code: only exit(1) when ALL items fail, not on partial failure

tests:

  • test_both_snapshots_fail_bidirectional: both adapters fail → FAILED
  • test_hf_snapshot_fails_ms_to_hf_ok: HF fails but MS_TO_HF only needs MS snapshot → SUCCESS (exposes _update_state bug that was fixed)

Summary by CodeRabbit

  • Bug Fixes
    • Improved snapshot error handling: sync continues when individual snapshot fetches fail in one-way sync, while bidirectional sync now fails if required snapshots are missing.
    • Exit code behavior adjusted: process now returns a failure exit when any sync item fails.
  • Tests
    • Added tests covering snapshot-fetch error scenarios and related exit behavior.

Review Change Stack

sync.yml:
- Fix word-splitting bug: replace string ARGS with bash array,
  same issue as entrypoint.sh (PR #1 review)

sync_engine.py:
- Wrap HF snapshot fetch in try/except (was unprotected, unlike MS)
- Add explicit guard: HF_TO_MS fails clearly when HF snapshot unavailable
- Add explicit guard: BIDIRECTIONAL fails clearly when both snapshots unavailable
- Fix _update_state: handle None hf_snapshot (was causing AttributeError)
- Change exit code: only exit(1) when ALL items fail, not on partial failure

tests:
- test_both_snapshots_fail_bidirectional: both adapters fail → FAILED
- test_hf_snapshot_fails_ms_to_hf_ok: HF fails but MS_TO_HF only
  needs MS snapshot → SUCCESS (exposes _update_state bug that was fixed)

Signed-off-by: dongjiang <dongjiang1989@126.com>
@coderabbitai

coderabbitai Bot commented May 29, 2026

Copy link
Copy Markdown

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2144950b-3cb9-446c-9a71-548c9af8a8e8

📥 Commits

Reviewing files that changed from the base of the PR and between 751f9c2 and 67fa193.

📒 Files selected for processing (2)
  • src/sync_engine.py
  • tests/test_sync_engine_extended.py

📝 Walkthrough

Walkthrough

This PR hardens snapshot fetch handling by wrapping HF snapshot retrieval in a try/except and validating required snapshots per sync mode, conditions HF state writes on snapshot presence, refactors CLI exit logic to fail on any FAILED item, updates the GitHub Actions step to pass args via a Bash array, and adds tests for these error paths.

Changes

Snapshot Fetch Resilience

Layer / File(s) Summary
Snapshot fetch error handling and validation
src/sync_engine.py
Initialize hf_snapshot = None, wrap get_repo_snapshot in try/except with warning logging; for BIDIRECTIONAL raise RuntimeError if either snapshot is missing; for HF_TO_MS raise RuntimeError if HF snapshot missing; only update hf_state.last_synced_commit when hf_snapshot exists.
Exit code logic update
src/sync_engine.py
Refactor CLI main() to compute any_failed boolean and call sys.exit(1) when any item result status is FAILED.
Workflow CLI argument passing
.github/workflows/sync.yml
Change sync invocation to build ARGS as a Bash array (init with --config and --state-dir), append conditional flags, and invoke python -m src.sync_engine "${ARGS[@]}".
Error handling test coverage
tests/test_sync_engine_extended.py
Add three tests: bidirectional with both HF/MS fetch failures -> FAILED; bidirectional with HF failure only -> FAILED with HF-specific message; ms_to_hf with HF failure but MS success -> SUCCESS.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

A rabbit nibbles through the stack,
hears snapshots stumble, then adapt,
warnings whisper, arrays align,
exit codes check each trailing line,
tests hop in to seal the pact. 🐰

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main changes: hardening the sync workflow with improved error handling for snapshot fetches and fixing word-splitting bugs.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-pre-dryrun-fail

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the sync engine to handle HuggingFace snapshot fetch failures gracefully by catching exceptions, logging warnings, and raising explicit errors during bidirectional or unidirectional syncs if required snapshots are missing. It also modifies the CLI exit behavior to only return a non-zero code if all sync items fail, and adds corresponding test cases. The review feedback highlights two important issues: first, a logical bug in the bidirectional check where using 'and' instead of 'or' can lead to an AttributeError if only one snapshot is missing; second, a recommendation to revert the partial-failure exit code change to prevent silent failures in CI/CD pipelines.

Comment thread src/sync_engine.py Outdated
Comment thread src/sync_engine.py Outdated
Address PR #4 review from gemini-code-assist:

1. BIDIRECTIONAL: change `and` to `or` in snapshot None check.
   Both snapshots are required for bidirectional sync — if either
   is None, detect_bidirectional() crashes with AttributeError on
   the None snapshot. Now raises RuntimeError with specific status
   per adapter (e.g. "HF=OK, MS=FAILED").

2. Exit code: revert to exit(1) on any failure (not just all-fail).
   CI/CD downstream steps already use `if: always()` so they run
   regardless of exit code. Returning 0 on partial failure would
   silently mask errors in automated pipelines.

+1 test: test_one_snapshot_fails_bidirectional

Signed-off-by: dongjiang <dongjiang1989@126.com>
@dongjiang1989
dongjiang1989 merged commit 971f83f into master May 29, 2026
10 of 11 checks passed
@dongjiang1989
dongjiang1989 deleted the fix-pre-dryrun-fail branch May 29, 2026 13:04
dongjiang1989 added a commit that referenced this pull request May 29, 2026
* fix: harden sync workflow and snapshot error handling (#3)

sync.yml:
- Fix word-splitting bug: replace string ARGS with bash array,
  same issue as entrypoint.sh (PR #1 review)

sync_engine.py:
- Wrap HF snapshot fetch in try/except (was unprotected, unlike MS)
- Add explicit guard: HF_TO_MS fails clearly when HF snapshot unavailable
- Add explicit guard: BIDIRECTIONAL fails clearly when both snapshots unavailable
- Fix _update_state: handle None hf_snapshot (was causing AttributeError)
- Change exit code: only exit(1) when ALL items fail, not on partial failure

tests:
- test_both_snapshots_fail_bidirectional: both adapters fail → FAILED
- test_hf_snapshot_fails_ms_to_hf_ok: HF fails but MS_TO_HF only
  needs MS snapshot → SUCCESS (exposes _update_state bug that was fixed)

Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix: bidirectional snapshot guard uses 'or', revert partial exit code

Address PR #4 review from gemini-code-assist:

1. BIDIRECTIONAL: change `and` to `or` in snapshot None check.
   Both snapshots are required for bidirectional sync — if either
   is None, detect_bidirectional() crashes with AttributeError on
   the None snapshot. Now raises RuntimeError with specific status
   per adapter (e.g. "HF=OK, MS=FAILED").

2. Exit code: revert to exit(1) on any failure (not just all-fail).
   CI/CD downstream steps already use `if: always()` so they run
   regardless of exit code. Returning 0 on partial failure would
   silently mask errors in automated pipelines.

+1 test: test_one_snapshot_fails_bidirectional

Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix: disk space, create_repo, and partial state persistence

Problem analysis from failed sync run (151s, exit code 1, empty .sync_state/):

1. Disk space: 4 parallel downloads of Qwen2.5-7B (~14GB total) fills
   up the GitHub Actions runner (~14GB). Fix: large files (>100MB)
   transfer sequentially (download→upload→delete→next), small files
   still use parallel.

2. create_repo_if_needed: was calling get_repo_snapshot (full file
   listing) just to check if repo exists. Fix: use repo_exists() which
   is a lightweight API call. Also wrap in try/except to be non-fatal.

3. _update_state: only called on full success, so partial transfers
   were lost. Fix: call _update_state whenever files_synced is non-empty,
   so successfully transferred files are recorded and won't be re-synced.

Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix: add repo_type to create_repo and cross-reference file maps in state update

PR #6 review fixes (2 of 3 comments valid):

1. modelscope_adapter.py: Pass repo_type parameter to create_repo()
   - Without this, dataset repos were incorrectly created as model repos
   - ModelScope SDK defaults repo_type='model' when not specified

2. sync_engine.py: Cross-reference file maps when updating sync state
   - When syncing MS→HF, the synced file's SHA is in ms_file_map, not hf_file_map
   - Without fallback, synced_files wouldn't be recorded in state
   - Now tries both maps: hf_file_map.get(fp) or ms_file_map.get(fp)
   - Same logic for MS state: ms_file_map.get(fp) or hf_file_map.get(fp)

Comment 1 (repo_exists method) was invalid: HubApi.repo_exists() does exist
in the ModelScope SDK (verified: returns bool, accepts repo_type parameter).

Signed-off-by: dongjiang <dongjiang1989@126.com>

---------

Signed-off-by: dongjiang <dongjiang1989@126.com>
dongjiang1989 added a commit that referenced this pull request May 29, 2026
Address PR #4 review from gemini-code-assist:

1. BIDIRECTIONAL: change `and` to `or` in snapshot None check.
   Both snapshots are required for bidirectional sync — if either
   is None, detect_bidirectional() crashes with AttributeError on
   the None snapshot. Now raises RuntimeError with specific status
   per adapter (e.g. "HF=OK, MS=FAILED").

2. Exit code: revert to exit(1) on any failure (not just all-fail).
   CI/CD downstream steps already use `if: always()` so they run
   regardless of exit code. Returning 0 on partial failure would
   silently mask errors in automated pipelines.

+1 test: test_one_snapshot_fails_bidirectional

Signed-off-by: dongjiang <dongjiang1989@126.com>
dongjiang1989 added a commit that referenced this pull request May 29, 2026
* fix: harden sync workflow and snapshot error handling (#3)

sync.yml:
- Fix word-splitting bug: replace string ARGS with bash array,
  same issue as entrypoint.sh (PR #1 review)

sync_engine.py:
- Wrap HF snapshot fetch in try/except (was unprotected, unlike MS)
- Add explicit guard: HF_TO_MS fails clearly when HF snapshot unavailable
- Add explicit guard: BIDIRECTIONAL fails clearly when both snapshots unavailable
- Fix _update_state: handle None hf_snapshot (was causing AttributeError)
- Change exit code: only exit(1) when ALL items fail, not on partial failure

tests:
- test_both_snapshots_fail_bidirectional: both adapters fail → FAILED
- test_hf_snapshot_fails_ms_to_hf_ok: HF fails but MS_TO_HF only
  needs MS snapshot → SUCCESS (exposes _update_state bug that was fixed)

Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix: bidirectional snapshot guard uses 'or', revert partial exit code

Address PR #4 review from gemini-code-assist:

1. BIDIRECTIONAL: change `and` to `or` in snapshot None check.
   Both snapshots are required for bidirectional sync — if either
   is None, detect_bidirectional() crashes with AttributeError on
   the None snapshot. Now raises RuntimeError with specific status
   per adapter (e.g. "HF=OK, MS=FAILED").

2. Exit code: revert to exit(1) on any failure (not just all-fail).
   CI/CD downstream steps already use `if: always()` so they run
   regardless of exit code. Returning 0 on partial failure would
   silently mask errors in automated pipelines.

+1 test: test_one_snapshot_fails_bidirectional

Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix: disk space, create_repo, and partial state persistence

Problem analysis from failed sync run (151s, exit code 1, empty .sync_state/):

1. Disk space: 4 parallel downloads of Qwen2.5-7B (~14GB total) fills
   up the GitHub Actions runner (~14GB). Fix: large files (>100MB)
   transfer sequentially (download→upload→delete→next), small files
   still use parallel.

2. create_repo_if_needed: was calling get_repo_snapshot (full file
   listing) just to check if repo exists. Fix: use repo_exists() which
   is a lightweight API call. Also wrap in try/except to be non-fatal.

3. _update_state: only called on full success, so partial transfers
   were lost. Fix: call _update_state whenever files_synced is non-empty,
   so successfully transferred files are recorded and won't be re-synced.

Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix: add repo_type to create_repo and cross-reference file maps in state update

PR #6 review fixes (2 of 3 comments valid):

1. modelscope_adapter.py: Pass repo_type parameter to create_repo()
   - Without this, dataset repos were incorrectly created as model repos
   - ModelScope SDK defaults repo_type='model' when not specified

2. sync_engine.py: Cross-reference file maps when updating sync state
   - When syncing MS→HF, the synced file's SHA is in ms_file_map, not hf_file_map
   - Without fallback, synced_files wouldn't be recorded in state
   - Now tries both maps: hf_file_map.get(fp) or ms_file_map.get(fp)
   - Same logic for MS state: ms_file_map.get(fp) or hf_file_map.get(fp)

Comment 1 (repo_exists method) was invalid: HubApi.repo_exists() does exist
in the ModelScope SDK (verified: returns bool, accepts repo_type parameter).

Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix(modelscope): improve error message for permission failures

When uploading to ModelScope fails due to lack of write access, the API
returns a misleading "resource does not exist" error. This commit adds
explicit error detection for permission failures (401/403/"does not exist")
and raises a clear PermissionError with actionable guidance:

- Tells the user they don't have write access
- Suggests using their own namespace (e.g., 'username/model-name')
- Applies to both SDK upload and HF fallback upload paths

This prevents confusing errors when users try to sync to repos they
don't own (e.g., 'Qwen/Qwen2.5-7B-Instruct' instead of their own repo).

Signed-off-by: dongjiang <dongjiang1989@126.com>

* chore(config): use own ModelScope namespace for sync target

The previous ms_repo_id 'Qwen/Qwen2.5-7B-Instruct' belongs to the
Qwen organization and requires write access that external users do
not have. Changed to 'dongjiang1989/Qwen2.5-7B-Instruct' so the
sync can create and upload to a repo under the user's own namespace.

Signed-off-by: dongjiang <dongjiang@kubeservice.com>
Signed-off-by: dongjiang <dongjiang1989@126.com>

* fix(modelscope): case-insensitive permission error detection

Use str(e).lower() when matching error messages so that variants
like "Does Not Exist", "Forbidden", or "FORBIDDEN" are all caught.
Also added "forbidden" as an additional keyword to match.

Addresses gemini-code-assist review on PR #8.

Signed-off-by: dongjiang <dongjiang@kubeservice.com>
Signed-off-by: dongjiang <dongjiang1989@126.com>

* style(modelscope): fix E501 line-too-long in permission checks

Refactor long `or` chain into `any()` with a tuple of keywords to
keep lines under 100 chars.

Signed-off-by: dongjiang <dongjiang@kubeservice.com>
Signed-off-by: dongjiang <dongjiang1989@126.com>

---------

Signed-off-by: dongjiang <dongjiang1989@126.com>
Signed-off-by: dongjiang <dongjiang@kubeservice.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sync failure on 2026-05-29

1 participant