Release 0.1.21: catalog/storage paths and Dremio catalog check in %metadata

README.md

@@ -36,7 +36,7 @@ pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"
 ```
 
 # staging's latest release (early access) — pin to the tag the "staging" badge above shows
-pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.20-staging"
+pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.21-staging"
 ```
 
 ## Usage
@@ -193,7 +193,11 @@ with SdiController.from_env() as sdi:
 
 - `get_xml(uuid)` checks the response is `mdb:MD_Metadata` and identifies `uuid`
   (`NotIso19115_3`, `UuidMismatch`; `SdiNotFound` / `SdiAuthError` for 404 / 401-403).
-- `push_to_dds(metadata, dds_path, folder="metadata", force=False)` compares with the copy
+- `push_to_dds(metadata, dds_path, folder="metadata", force=False)` takes `dds_path` in catalog
+  format (`a.b.c`, sent to DDS as `a/b/c`; quoted names like `"v1.0"` lose their quotes and stay
+  one segment) or storage format (`a/b/c`). It first checks `dds_path` exists in the Dremio
+  catalog when the controller has a catalog client (`CatalogPathNotFound` otherwise), then
+  compares with the copy
   already in DDS: same bytes is `unchanged`; an older copy is `replaced`; a copy that is newer or
   undated (someone edited it in DDS) raises `DdsCopyConflict` unless `force=True`.
 - `resolve_series(series_uuid)` returns the one release of a series that is not superseded, or
@@ -220,9 +224,13 @@ import eea_datalakehouse.notebook   # registers %catalog/%ingest/%sdi/%metadata
 %metadata push_to_dds("catalog/water_management_resources/bathing_water/bwd")
 ```
 
-For the push, `DDS_BASE_URL` is read from the first `.env` in the notebook's folder or a parent
-when `%metadata` builds its session (the kernel environment's value only applies if no `.env`
-sets it; `%metadata dds_base_url()` shows which is in use). The Dremio identity is `_DREMIO_USER` /
+For the push, `DDS_BASE_URL` and `DREMIO_BASE_URL` are read from the first `.env` in the
+notebook's folder or a parent when `%metadata` builds its session (the kernel environment's
+value only applies if that `.env` doesn't set one; `%metadata dds_base_url()` /
+`dremio_base_url()` show which is in use). `%metadata push_to_dds` first checks that `dds_path`
+exists in the Dremio catalog (a `/` path is converted to `a.b.c` for that) and uploads nothing
+if it doesn't; `%metadata check_catalog_path(path)` runs the check alone, and
+`check_catalog=False` skips it. The Dremio identity is `_DREMIO_USER` /
 `_DREMIO_PWD` (as `%ingest`), falling back to `DREMIO_USERNAME` / `DREMIO_TOKEN` (as `%catalog`).
 
 ## Notebook facade (`%catalog` / `%ingest`)
@@ -419,7 +427,7 @@ To pin to one specific release instead, use the exact tag the live badges under
 %pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.19"
 
 # staging's latest release (early access)
-%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.20-staging"
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.21-staging"
 ```
 
 Use the `%pip` magic rather than `!pip` — it installs into the kernel the

debugger/sdi_session_example.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# `%sdi` + `%metadata` — SDI metadata into DDS\n\nA test notebook for the `%sdi` and `%metadata` magics from `eea_datalakehouse.notebook.magics`.\n`%sdi` downloads a dataset's ISO 19115-3 metadata record from the EEA SDI catalogue;\n`%metadata` uploads it to the dataset's `metadata` folder in DDS — a **document upload**, never an ingest: nothing becomes a\nDremio table. See `docs/sdi-integration-plan.md` for the design.\n\n**Kernel environment** — read the same way `%catalog` and `%ingest` read theirs, when each magic's\nfirst call builds its session:\n\n| Variable | Needed for | Notes |\n|---|---|---|\n| `SDI_API_URL` | `%sdi get_xml`, `%sdi resolve_series` | empty or unset → `https://sdi.eea.europa.eu/catalogue` |\n| `SDI_USERNAME` / `SDI_PASSWORD` | non-public records only | optional |\n| `DDS_BASE_URL` | `%metadata push_to_dds` | read from the first `.env` in the notebook's folder or a parent (here `debugger/.env`); else the kernel environment |\n| `_DREMIO_USER` / `_DREMIO_PWD` | `%metadata push_to_dds` | as for `%ingest`; else `DREMIO_USERNAME` / `DREMIO_TOKEN`, as for `%catalog` |\n\n`%sdi` makes public, read-only calls and needs none of the DDS settings. Install the extra this\nneeds once: `pip install \"EEADataLakehouse[notebook]\"`.",
+   "source": "# `%sdi` + `%metadata` — SDI metadata into DDS\n\nA test notebook for the `%sdi` and `%metadata` magics from `eea_datalakehouse.notebook.magics`.\n`%sdi` downloads a dataset's ISO 19115-3 metadata record from the EEA SDI catalogue;\n`%metadata` uploads it to the dataset's `metadata` folder in DDS — a **document upload**, never an ingest: nothing becomes a\nDremio table. See `docs/sdi-integration-plan.md` for the design.\n\n**Kernel environment** — read the same way `%catalog` and `%ingest` read theirs, when each magic's\nfirst call builds its session:\n\n| Variable | Needed for | Notes |\n|---|---|---|\n| `SDI_API_URL` | `%sdi get_xml`, `%sdi resolve_series` | empty or unset → `https://sdi.eea.europa.eu/catalogue` |\n| `SDI_USERNAME` / `SDI_PASSWORD` | non-public records only | optional |\n| `DDS_BASE_URL` | `%metadata push_to_dds` | read from the first `.env` in the notebook's folder or a parent (here `debugger/.env`); else the kernel environment |\n| `DREMIO_BASE_URL` | `%metadata push_to_dds`, `check_catalog_path` | the catalog check; read from `.env` first, like `DDS_BASE_URL` |\n| `_DREMIO_USER` / `_DREMIO_PWD` | `%metadata push_to_dds` | as for `%ingest`; else `DREMIO_USERNAME` / `DREMIO_TOKEN`, as for `%catalog` |\n\n`%sdi` makes public, read-only calls and needs none of the DDS settings. Install the extra this\nneeds once: `pip install \"EEADataLakehouse[notebook]\"`.",
    "id": "bc3d7aa0"
   },
   {
@@ -39,15 +39,15 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "## Settings for this test\n\nThe bathing water release used throughout the library's tests. The DDS path is\n**provisional** until the DDS document path convention is agreed; `%metadata push_to_dds` appends\n`/metadata/<uuid>.xml` to it.",
+   "source": "## Settings for this test\n\nThe bathing water release used throughout the library's tests. The DDS path is\n**provisional** until the DDS document path convention is agreed; `%metadata push_to_dds` appends\n`/metadata/<uuid>.xml` to it.\n\n`DDS_PATH` can be given in catalog format (`a.b.c`, as `%catalog` paths are written) or in\ndata storage format (`a/b/c`). A catalog path has its `.` separators replaced with `/` before\nit is sent to the DDS REST API; a path that already contains `/` is split on `/` instead.\nQuoted names (`catalog.\"bathing water\".\"v1.0\"`, as Dremio SQL writes them) lose their quotes\nand stay one segment, so the path matches the folders that exist in the Dremio catalog.",
    "id": "200b71fe"
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": "RELEASE_UUID = \"070d9baa-448d-4168-8514-7dadb3ad876d\"   # Bathing Water Directive, 2025 v1.0\nSERIES_UUID = \"c3858959-90da-4c1b-b9ca-492db0e514df\"    # its series\nDDS_PATH = \"catalog/water_management_resources/bathing_water/bwd\"",
+   "source": "RELEASE_UUID = \"070d9baa-448d-4168-8514-7dadb3ad876d\"   # Bathing Water Directive, 2025 v1.0\nSERIES_UUID = \"c3858959-90da-4c1b-b9ca-492db0e514df\"    # its series\nDDS_PATH = \"catalog.water_management_resources.bathing_water.bwd\"   # sent as catalog/water_management_resources/...",
    "id": "fb073e8a"
   },
   {
@@ -120,6 +120,28 @@
    "outputs": [],
    "source": "%metadata dds_base_url()"
   },
+  {
+   "cell_type": "code",
+   "id": "8556848d",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%metadata dremio_base_url()"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e036a34b",
+   "metadata": {},
+   "source": "`push_to_dds` refuses to upload unless `DDS_PATH` (without the `metadata` folder) already exists in the\nDremio catalog. Check it on its own first — a read-only lookup; a path written with `/` is\nconverted to the catalog path structure (`a.b.c`) for it:"
+  },
+  {
+   "cell_type": "code",
+   "id": "eec1adb8",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%metadata check_catalog_path(DDS_PATH)"
+  },
   {
    "cell_type": "code",
    "execution_count": null,

docs/sdi-integration-plan.md

@@ -275,10 +275,17 @@ per kernel each, built on first use from the kernel environment:
   (looked up at push time) unless `metadata=` is given.
 - Every failure prints one `sdi error: …` / `metadata error: …` line instead
   of a traceback.
-- `DDS_BASE_URL` is read, when `%metadata` builds its session, from the first
-  `.env` in the notebook's folder or a parent; the kernel environment's value
-  applies only if no `.env` sets it. `%metadata dds_base_url()` shows which is
-  in use.
+- `DDS_BASE_URL` and `DREMIO_BASE_URL` are read, when `%metadata` builds its
+  session, from the first `.env` in the notebook's folder or a parent; the
+  kernel environment's value applies only if that `.env` doesn't set one.
+  `%metadata dds_base_url()` / `dremio_base_url()` show which is in use.
+- **`push_to_dds` checks the Dremio catalog first**: `dds_path` (without the
+  `metadata` folder) must exist there, looked up with Dremio's
+  `/api/v3/catalog/by-path` and the same PAT; otherwise nothing is uploaded
+  (`CatalogPathNotFound`). A path with `/` separators is converted to the
+  catalog path structure (`a.b.c`, quoting names that need it) for the check
+  and the messages. `%metadata check_catalog_path(path)` runs the check alone;
+  `check_catalog=False` skips it.
 - The DDS upload's Dremio identity is `_DREMIO_USER` / `_DREMIO_PWD` (as
   `%ingest`), falling back to `DREMIO_USERNAME` / `DREMIO_TOKEN` (as
   `%catalog`).

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "EEADataLakehouse"
-version = "0.1.20"
+version = "0.1.21"
 description = "Data preparation and validation utilities for the EEA data lakehouse pipeline."
 readme = "README.md"
 requires-python = ">=3.11"

src/eea_datalakehouse/catalog/rest.py

@@ -17,8 +17,9 @@
 
 from __future__ import annotations
 
-from collections.abc import Callable
+from collections.abc import Callable, Sequence
 from typing import Any
+from urllib.parse import quote
 
 import httpx
 
@@ -58,7 +59,24 @@ def close(self) -> None:
 
     def _lookup_by_path(self, path: str) -> dict[str, Any] | None:
         """The catalog entity's own JSON (id, type, ...) at `path`, or None if absent."""
-        encoded = "/".join(path.split("."))
+        return self._lookup_encoded("/".join(path.split(".")), path)
+
+    def get_entity(self, segments: Sequence[str]) -> dict[str, Any] | None:
+        """The catalog entity's JSON at the path made of `segments`, or None if absent.
+
+        One item per path level, each a name exactly as it appears in the
+        catalog — unlike the dot-separated `path` every other method takes, a
+        name here may itself contain dots, spaces or brackets (each segment is
+        URL-encoded on its own). Not best-effort: a lookup failure other than
+        404 raises `CatalogOperationError` (or `EngineStartingError` on a
+        timeout) rather than reading as "absent".
+        """
+        if not segments:
+            raise CatalogOperationError("cannot look up an empty catalog path")
+        encoded = "/".join(quote(segment, safe="") for segment in segments)
+        return self._lookup_encoded(encoded, ".".join(segments))
+
+    def _lookup_encoded(self, encoded: str, path: str) -> dict[str, Any] | None:
         try:
             resp = self._http.get(
                 f"{self._base_url}/api/v3/catalog/by-path/{encoded}", headers=self._headers

src/eea_datalakehouse/notebook/magics.py

@@ -110,8 +110,8 @@
     ),
     "sdi": '%sdi get_xml("<uuid>")  |  %sdi resolve_series("<series uuid>")',
     "metadata": (
-        '%metadata push_to_dds("a/b/c")  (uploads the last %sdi get_xml result)'
-        "  |  %metadata dds_base_url()"
+        '%metadata push_to_dds("a.b.c")  or  %metadata push_to_dds("a/b/c")'
+        "  (uploads the last %sdi get_xml result)"
     ),
 }
 
@@ -265,25 +265,35 @@
     "Environment: SDI_API_URL (empty means the public EEA catalogue; optional "
     "SDI_USERNAME/SDI_PASSWORD for non-public records). Pushing to DDS is %metadata's job."
 )
+# dds_base_url()/dremio_base_url() still work under %metadata; they are left
+# out of the help table on purpose, as troubleshooting helpers rather than
+# everyday commands.
 _METADATA_HELP = [
     (
         "push_to_dds",
         "Upload metadata (default: the last %sdi get_xml result) to "
-        "dds_path/folder/<uuid>.xml in DDS — a document upload, never an ingest. Same bytes "
-        "already there: unchanged; an older copy: replaced; a copy edited in DDS: refused "
-        "unless force=True.",
+        "dds_path/folder/<uuid>.xml in DDS — a document upload, never an ingest. dds_path "
+        "must already exist in the Dremio catalog (check_catalog=False skips that check). "
+        "Same bytes already there: unchanged; an older copy: replaced; a copy edited in DDS: "
+        "refused unless force=True.",
     ),
     (
-        "dds_base_url",
-        "Show the DDS URL push_to_dds uses and where it came from (a .env file or the kernel "
-        "environment).",
+        "check_catalog_path",
+        "Check dds_path exists in the Dremio catalog without pushing anything.",
     ),
 ]
 _METADATA_HELP_NOTE = (
-    "Environment: DDS_BASE_URL — read from the first .env found in the notebook's folder "
-    "or a parent when %metadata starts, else from the kernel environment — and a Dremio "
-    "identity: _DREMIO_USER/_DREMIO_PWD (as %ingest) or DREMIO_USERNAME/DREMIO_TOKEN "
-    "(as %catalog)."
+    "dds_path is the dataset's folder path, without the metadata folder. Its folders can be "
+    "separated with either '.' or '/', so these are the same path:\n"
+    "  catalog.water_management_resources.bathing_water.bwd\n"
+    "  catalog/water_management_resources/bathing_water/bwd\n"
+    "A name containing a '.' or a space is written in double quotes with '.' separators "
+    '(catalog."bathing water"."v1.0"), or as it is with \'/\' separators '
+    "(catalog/bathing water/v1.0).\n"
+    "Environment: DDS_BASE_URL and DREMIO_BASE_URL — read from the first .env found in the "
+    "notebook's folder or a parent when %metadata starts, else from the kernel environment "
+    "— and a Dremio identity: _DREMIO_USER/_DREMIO_PWD (as %ingest) or "
+    "DREMIO_USERNAME/DREMIO_TOKEN (as %catalog), used for both DDS and the catalog check."
 )
 
 

src/eea_datalakehouse/sdi/__init__.py

@@ -22,8 +22,12 @@
     SdiController,
     SdiMetadata,
     metadata_path,
+    path_segments,
+    to_catalog_path,
+    to_storage_path,
 )
 from .errors import (
+    CatalogPathNotFound,
     DdsCopyConflict,
     NotCurrentError,
     NotIso19115_3,
@@ -38,6 +42,7 @@
 from .session import MetadataSession, MetadataSessionError, SdiSession, SdiSessionError
 
 __all__ = [
+    "CatalogPathNotFound",
     "DEFAULT_FOLDER",
     "DEFAULT_SDI_API_URL",
     "DdsCopyConflict",
@@ -60,4 +65,7 @@
     "SeriesCandidate",
     "UuidMismatch",
     "metadata_path",
+    "path_segments",
+    "to_catalog_path",
+    "to_storage_path",
 ]