Skip to content

Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements#5

Draft
bkeepers wants to merge 1 commit into
mainfrom
aca-ncaribbean-sdb
Draft

Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements#5
bkeepers wants to merge 1 commit into
mainfrom
aca-ncaribbean-sdb

Conversation

@bkeepers

@bkeepers bkeepers commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Ingests Allen Coral Atlas 10 m satellite-derived bathymetry (SDB) for the Northern
Caribbean / Bahamas as sources/aca_ncaribbean (CC-BY 4.0; raw mirrored to R2), plus several
general multi-source pipeline improvements. The ACA dataset is being tabled — the recipe
runs end-to-end, but SDB isn't good enough for a nautical chart. This PR preserves the work and
documents the challenges for a later revisit. Not intended to merge as-is (keeping it on a
branch leaves main's build untouched).

Pipeline improvements (keep — independently useful)

  • tile_raster.py — split an oversized source raster into memory-safe per-tile rasters
    (windowed gdal_translate; drops all-nodata tiles). Also the tool for the deferred GSC Pacific ingest.
  • source_smooth.py — gaussian denoise over water for speckly sources; optional --max-depth cutoff.
  • source_datum.py--scale (unit conversion applied before negate; e.g. ACA cm→m).
  • source_polygonize.py — generalize large/speckly coverage masks before polygonizing
    (pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit).
  • aggregation_run.py — removed SKIP_CONTOURS / SKIP_SMOOTH; aggregation stages
    (reproject/merge/smooth/tile/contour) now run independently via a step <name>... CLI instead
    of being skipped by env flags.
  • Priority quality tiersnoaa_s102=2, cudem/cudem_third=1 (real surveys outrank
    lower-confidence sources in overlap).

Why ACA is tabled — the challenges

Research confirmed ACA is the only open, dense, redistributable bathymetry for the Bahamas banks
(the Andros "Median DEM" is paywalled; ICESat-2 ATL24 is open but sparse along-track; TCarta/EOMAP
are proprietary). But passive-optical SDB is fundamentally hard for a safety chart:

  1. Vertical noise. ±1–2 m with per-pixel speckle. It needed ~σ8 (≈80 m) gaussian smoothing to be
    usable — i.e. its honest resolution is ~z11 (~69 m), not the nominal 10 m.
  2. False-shallow over deep water. ACA returns shallow (<18 m) values over water GEBCO shows is
    1,500–4,800 m deep. A self-referential depth cutoff can't catch this (the noise reads shallow).
    A GEBCO-depth-gated feather (w=1 where GEBCO ≥ −12 m, ramp → 0 by −25 m; out = w·ACA + (1−w)·GEBCO)
    does fix the terrain (deep→GEBCO, shallow→ACA) and is a reusable building block.
  3. Contour explosion (the blocker). Even after the gate, contouring the vast shallow banks at the
    pipeline's fine 1 m intervals yields ~1 M features per macrotile; tippecanoe takes 45 min+
    to tile that (a global build would be untenable). The only lever that cuts the count is coarsening the
    shallow interval — but fine-grained shallow contours are essential for navigation, so that's not
    acceptable. Neither σ nor the GEBCO gate reduces the feature count.

Revisit path

  • Ingest ICESat-2 ATL24 (open, NASA public domain, RMSE ~0.3 m) as sparse depth control / QA,
    and/or to bias-correct ACA.
  • Or re-derive SDB with C-SHELPh (open) on ICESat-2 + Sentinel-2 for a cleaner, de-speckled owned grid.
  • The GEBCO-gated feather (fixes the terrain) is the reusable piece either way.

See the ROADMAP.md Caribbean table for the tabled entry.

…eline improvements

General pipeline (independently useful):
- tile_raster.py: split an oversized source raster into memory-safe per-tile rasters
  (windowed gdal_translate; drops all-nodata tiles)
- source_smooth.py: gaussian denoise over water for speckly sources (+ optional --max-depth cutoff)
- source_datum.py: --scale for unit conversion (e.g. cm->m), applied before negate
- source_polygonize.py: generalize large/speckly coverage masks before polygonizing
  (pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit)
- aggregation_run.py: remove SKIP_CONTOURS/SKIP_SMOOTH; stages run independently via a `step` CLI
- source priority quality tiers: noaa_s102=2, cudem/cudem_third=1

sources/aca_ncaribbean (ACA 10 m SDB, CC-BY 4.0): recipe complete end-to-end and the raw
GeoTIFF is mirrored to R2, but the dataset is TABLED -- SDB is too noisy for a nautical
chart (see PR for the write-up). Not wired into a shippable build; branch kept for revisit.
@bkeepers bkeepers marked this pull request as draft July 1, 2026 17:36
@bkeepers bkeepers changed the title Table Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant