Skip to content

Adopt the shared versioned crawl-result contract #407

Description

@KingAkeem

Why

GoTor already emits a versioned integration-safe JSON crawl report and exposes a documented job-control API. TorBot still has separate legacy result shapes and prior closed requests for JSON/save behavior, which makes the related crawler stack harder to integrate and test consistently.

Adopting one versioned crawl-result and run-evidence contract would reduce duplicate serialization logic across TorBot, GoTor, and desktop consumers while preserving each repository's implementation boundaries.

Proposed scope

Use GoTor's versioned report as the starting contract, document the compatibility decisions, and add a TorBot adapter that emits the shared shape.

The shared shape should include:

  • schema and run identifiers;
  • normalized target and effective crawl settings;
  • backend/component version;
  • per-page outcome, status, skip reason, and bounded error category;
  • extracted link/contact provenance;
  • cancellation and terminal status;
  • start/end timestamps and duration;
  • privacy-safe diagnostics without credentials, cookies, or unnecessary sensitive content.

Acceptance criteria

  • Document the canonical versioned crawl-result schema and compatibility rules.
  • Add a TorBot serializer/adapter for the shared schema without breaking existing human-readable CLI output.
  • Add fixtures covering success, partial page failures, skipped content, cancellation, and invalid input.
  • Add contract tests that compare TorBot output with representative GoTor report fixtures.
  • Document how TorBotApp or another consumer can feature-detect schema versions.
  • Define a deprecation path for overlapping legacy JSON/save formats instead of silently changing them.
  • Ensure errors and provenance remain useful without persisting secrets or unrestricted page contents.
  • Link or update the relevant integration documentation in both projects where maintainers have access.

Existing-work boundaries

  • Do not duplicate the workflow modernization tracked in Modernize Python CI matrix and GitHub Actions #404.
  • Do not reopen the old generic save/JSON issues; this issue is specifically about a shared, versioned cross-repository contract.
  • Do not couple TorBot directly to GoTor internals. Share the contract and fixtures, not application code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions