Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements#5
Draft
bkeepers wants to merge 1 commit into
Draft
Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements#5bkeepers wants to merge 1 commit into
bkeepers wants to merge 1 commit into
Conversation
…eline improvements General pipeline (independently useful): - tile_raster.py: split an oversized source raster into memory-safe per-tile rasters (windowed gdal_translate; drops all-nodata tiles) - source_smooth.py: gaussian denoise over water for speckly sources (+ optional --max-depth cutoff) - source_datum.py: --scale for unit conversion (e.g. cm->m), applied before negate - source_polygonize.py: generalize large/speckly coverage masks before polygonizing (pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit) - aggregation_run.py: remove SKIP_CONTOURS/SKIP_SMOOTH; stages run independently via a `step` CLI - source priority quality tiers: noaa_s102=2, cudem/cudem_third=1 sources/aca_ncaribbean (ACA 10 m SDB, CC-BY 4.0): recipe complete end-to-end and the raw GeoTIFF is mirrored to R2, but the dataset is TABLED -- SDB is too noisy for a nautical chart (see PR for the write-up). Not wired into a shippable build; branch kept for revisit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ingests Allen Coral Atlas 10 m satellite-derived bathymetry (SDB) for the Northern
Caribbean / Bahamas as
sources/aca_ncaribbean(CC-BY 4.0; raw mirrored to R2), plus severalgeneral multi-source pipeline improvements. The ACA dataset is being tabled — the recipe
runs end-to-end, but SDB isn't good enough for a nautical chart. This PR preserves the work and
documents the challenges for a later revisit. Not intended to merge as-is (keeping it on a
branch leaves
main's build untouched).Pipeline improvements (keep — independently useful)
tile_raster.py— split an oversized source raster into memory-safe per-tile rasters(windowed
gdal_translate; drops all-nodata tiles). Also the tool for the deferred GSC Pacific ingest.source_smooth.py— gaussian denoise over water for speckly sources; optional--max-depthcutoff.source_datum.py—--scale(unit conversion applied before negate; e.g. ACA cm→m).source_polygonize.py— generalize large/speckly coverage masks before polygonizing(pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit).
aggregation_run.py— removedSKIP_CONTOURS/SKIP_SMOOTH; aggregation stages(reproject/merge/smooth/tile/contour) now run independently via a
step <name>...CLI insteadof being skipped by env flags.
noaa_s102=2,cudem/cudem_third=1(real surveys outranklower-confidence sources in overlap).
Why ACA is tabled — the challenges
Research confirmed ACA is the only open, dense, redistributable bathymetry for the Bahamas banks
(the Andros "Median DEM" is paywalled; ICESat-2 ATL24 is open but sparse along-track; TCarta/EOMAP
are proprietary). But passive-optical SDB is fundamentally hard for a safety chart:
usable — i.e. its honest resolution is ~z11 (~69 m), not the nominal 10 m.
1,500–4,800 m deep. A self-referential depth cutoff can't catch this (the noise reads shallow).
A GEBCO-depth-gated feather (
w=1where GEBCO ≥ −12 m, ramp → 0 by −25 m;out = w·ACA + (1−w)·GEBCO)does fix the terrain (deep→GEBCO, shallow→ACA) and is a reusable building block.
pipeline's fine 1 m intervals yields ~1 M features per macrotile; tippecanoe takes 45 min+
to tile that (a global build would be untenable). The only lever that cuts the count is coarsening the
shallow interval — but fine-grained shallow contours are essential for navigation, so that's not
acceptable. Neither σ nor the GEBCO gate reduces the feature count.
Revisit path
and/or to bias-correct ACA.
See the
ROADMAP.mdCaribbean table for the tabled entry.