Skip to content

Python CLI setup round trip collapses dotted namespace levels #5291

Description

@mattfaltyn

Describe the bug

polaris setup export serializes each namespace as a dot-delimited string, and
polaris setup apply splits that string on every dot. A dot is valid inside an
individual Iceberg namespace level, so the distinct namespaces
["finance.us"] and ["finance", "us"] both export as finance.us and both
apply as the nested namespace ["finance", "us"].

This makes setup exports lossy for valid catalogs and contradicts the documented
export/apply backup and migration contract. The same encoding is also used for
namespace-scoped privileges and policy namespaces, so those references can be
restored against the wrong namespace identity.

To Reproduce

  1. Create a catalog containing the single-level namespace ["finance.us"]
    through the Iceberg REST API or another client that preserves namespace
    levels.
  2. Run polaris setup export > setup.yaml.
  3. Apply the result to an empty environment with
    polaris setup apply setup.yaml.
  4. List namespaces in the target catalog.

A deterministic unit-level reproduction against current main produces:

source=['finance.us'] exported=[{'name': 'finance.us'}]
applied=[['finance'], ['finance', 'us']]

source=['finance', 'us'] exported=[{'name': 'finance.us'}]
applied=[['finance'], ['finance', 'us']]

The first case is the failure. The second is a control showing that two distinct
source identities collide in the exported representation.

Actual Behavior

The source namespace ["finance.us"] is not recreated. Apply creates the parent
["finance"] and nested namespace ["finance", "us"] instead. Policies and
namespace grants using the same dotted encoding can likewise target the nested
namespace rather than the original single-level namespace.

Expected Behavior

Setup export/apply should preserve namespace levels exactly, keeping
["finance.us"] distinct from ["finance", "us"], while continuing to accept
existing dot-delimited setup files for backward compatibility.

Additional context

The Iceberg REST schema models a namespace as an array of string levels, and
Polaris's entity-name validation allows dots inside a level. At
ffa69775ec832a2e8308e3f75ffbc9b2d556c472:

  • _export_namespaces_for_catalog uses ".".join(ns).
  • _create_namespaces uses ns_name.split(".").
  • Namespace privilege export/apply and policy export/apply repeat the same
    lossy conversion.

I searched open and closed issues and pull requests across all states using
setup-export, setup-apply, dotted/literal-dot namespace, namespace levels,
separator, delimiter, serialization, collision, and round-trip terms. I also
inspected every currently open or draft PR touching setup.py or its tests.
Existing setup work covers nested traversal, policy-name collisions,
principal-role properties, storage fields, and terminal rendering, but not this
namespace-level identity collision.

This is a non-security correctness and backup/migration reliability issue.

System information

  • OS: macOS 26.6.1, arm64
  • Polaris Catalog Version: main at
    ffa69775ec832a2e8308e3f75ffbc9b2d556c472
  • Python: 3.13.13
  • Storage/setup: mocked API-transport reproduction; storage backend independent

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions