Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE-3182] Feat/core 3182/schema registry json external references #23677

Draft
wants to merge 10 commits into
base: dev
Choose a base branch
from

Commits on Oct 13, 2024

  1. schema_registry/avro: move collect_schema to util.h

    no change in behavior, will be used for external references for json
    schema
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    ab129a9 View commit details
    Browse the repository at this point in the history
  2. schema_registry/util: collect_schema overlad, collected_schema::get

    collect_schema overload accepts a list of references directly and does
    not save the input schema in the collected_schema list
    
    colleted_schema::get extranct the map of reference_name ->
    reference_iobuf
    to implement this, a bit of refactoring is applied to the class
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    74ede2a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    439ab9a View commit details
    Browse the repository at this point in the history
  4. schema_registry/json: prepare document_context to store external schemas

    no behavior change in this commit
    
    add a map external_schema_name -> json::Document, change the
    mapped_type of schemas_index to an union of local_ptr (json pointer to
    the root schema) and external_ptr (ptr to a named external schema in
    .external_schemas map)
    
    the type is templated to support rapidjson and jsoncons, this will be
    used later
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    1e13ae5 View commit details
    Browse the repository at this point in the history
  5. schema_registry/json: add the support to resolve external references

    this is done with schema_context::resolve_reference. the function will
    get an uri and return a json::Value::ConstObject or throw.
    
    the search is done with schema_index, a local_ptr will be resolved
    against the root schema and an external_ptr will resolve against one of
    the external_schemas.
    
    note that there is no behavior change yet, because the external_schemas
    map is not populated yet
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    e746327 View commit details
    Browse the repository at this point in the history
  6. schema_registry/json: convert parse_json to coroutine

    no behavior change
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    036285d View commit details
    Browse the repository at this point in the history
  7. schema_registry/json: move parse_json to jsoncons-only

    this is done to reduce conversions when parse_json will call itself to
    handle external references.
    
    the name change to parse_jsoncons, and a new parse_json calls the former
    and applies the translation to rapidjson of the result.
    
    collect_bundled_schema_and_fix_ref has to change too.
    
    no behavior change yet
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    793efc5 View commit details
    Browse the repository at this point in the history
  8. schema_registry/json: parse_json extract external schemas

    this commit adds support for external schemas: parse_json gets an
    (optional) id for the root schema, and a list of external schemas name
    to build into the result.
    
    each external schema il collected, parse_jsoncons invoked on it, and
    it's list of bundles schemas is added to the root schemas_index.
    
    each bundled schema has to have an unique id (due to the current
    resolve_reference implementation) and the external_schema.$id has to be
    unique (if it's not present, the exernal schema name will be used as
    $id)
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    93ade2a View commit details
    Browse the repository at this point in the history
  9. schema_registry/json collect_bundled_schemas_and_fix_refs improvement

    - if we are visiting the root, skip validation as it's already validated
      externally
    
    - if we are visiting a bundled schema, ensure its $id is unique
    andijcr committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    c8c8da9 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. schema_registry/json: remaning work

    this commit is a wip an unlikely to be finished.
    
    the idea is to add $ref validation at parse time, and at the same time
    handle cleanily various edge cases of external schemas
    
    to do so:
    1. external refs are changed to
       https://schema-registry.com/extrnal_ref_name
       this is to make it work like a bundled ref, in resolve_ref
       note: this does not handle well refs/with/slashes in the middle. they
       look like a valid uri path. try to add a test with them and see what
       breaks or not
    2. at collect_bundled_schemas_and_fix_ref, first do a tree traversal for
       id, and save refs in an output parameter unresolved_ref.
       after the first run, check every ref: if it's not a bundled/local
       ref, or an external ref, throw. bundled/local ref are made absolute,
       external refs are encoded as (1)
       a new parameter controls what to do if a ref is unknown. for the root
       schema, an unknown ref should raise an error. for an external schema,
       assume that it's external, because parse_jsocons is recursive
       and gets invoked for each external schema to extract
       their bundled schemas index.
       it is assumed that external schemas are already good (no unresolved
       refs) because they are already accepted, so we can limit the
       recursion to 2 levels (root, and then one for each direct/indirect
       external schema)
    3. schema_context::resolve_reference does not need to handle external
       schemas anymore, since they appear like bundled schemas (the uri is
       either in the index or not). that code can be simplified
    andijcr committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    821ab2e View commit details
    Browse the repository at this point in the history