Repository navigation
feat!: typed errors, a Stream-based streaming API, and a narrower public API - #65
Merged
Merged
Conversation
tycho
added this pull request to stack #66
October 5, 2026 18:03
tycho
force-pushed
the
steven/feat/typed-api
branch
from
October 5, 2026 18:06
13206f6 to
cc8ce8b
Compare
Base automatically changed from
steven/feat/manifest-json-image-manifest
to
main
October 5, 2026 18:10
Every module was public, including the merge engine's internals (`canonical`, `overlay`, `tracker`, `layers`) and the per-format output sinks (`dir`, `squashfs`, `tar`), even though their own documentation describes them as pipeline internals. All of it was API to keep stable, and the move to a typed error would have had to cover every one. Keep public only what consumers use: the conversion API at the crate root, `verify`, and `image` for parsing image layouts (the CLI's tests check its fetched layouts with `image::load_manifest`). `LayerBlob` is also re-exported at the crate root, where the streaming API uses it. The output sinks' convenience wrappers that nothing calls any more are removed, and the batch merge and `CanonicalTarHeader::path`, which only the tests use, are compiled for tests only. The tests that exercise internal modules move from `tests/` into the crate, as unit tests under `src/tests/`. BREAKING CHANGE: the `canonical`, `dir`, `layers`, `overlay`, `squashfs`, `tar` and `tracker` modules are no longer public; use the conversion API at the crate root instead. Signed-off-by: Steven Noonan <steven@edera.dev>
Every fallible function returned `anyhow::Error`, which suits an application but not a library: a caller could only print the error or guess at downcasts, could not tell a corrupt layer from a missing mksquashfs or a failed download, and was pushed into anyhow itself (`StreamingPacker::notify_error` took an `anyhow::Error`). Return `ocirender::Error`, a non-exhaustive enum built with thiserror whose variants say what failed: `ImageLayout`, `Layer` (with the layer's index and blob), `LayerSource` (the caller's own error, as its source), `MissingLayers`, `LayerIndexOutOfRange`, `PackerStopped`, `MksquashfsSpawn`, `Mksquashfs` (with its stderr), `Output` and `Verify`. As is conventional, `Display` describes only that level and `source()` carries the cause. `notify_error` and the streaming channels take any error as a `BoxError`, so a caller's own error type, anyhow's included, still passes straight through. Typing the errors meant deciding what to blame when a merge stops, which the old code muddled. A corrupt layer usually surfaces while its data is being copied to the sink, where `tar::Builder` reports a read failure, a write failure and an entry it cannot encode as the same kind of `io::Error`. The sink is now wrapped to record its own failures, so only those are output errors and anything else is the layer's. With that, when mksquashfs dies its failure and stderr are reported rather than the broken pipe the merge then hits, while a corrupt layer or a failed download still takes precedence over the mksquashfs exit it causes. Along the way: the mksquashfs output path is passed as an `OsStr`, dropping a needless UTF-8 requirement; a panic in the blocking conversion task is resumed in the caller rather than turned into an error; and `image::strip_digest_prefix` returns an `Option`, so that the error for an unsupported digest names the file it appeared in. BREAKING CHANGE: fallible functions return `ocirender::Result` with `ocirender::Error` instead of `anyhow::Result`. `notify_error` and the streaming receivers take a `BoxError` (anything converting into `Box<dyn Error + Send + Sync>`, including `anyhow::Error`), and `image::strip_digest_prefix` returns `Option<&str>`. Signed-off-by: Steven Noonan <steven@edera.dev>
The streaming merge counted every layer it was handed towards the image's total without checking the layer's index. A layer with an index the image doesn't have was buffered and never processed, and a layer delivered twice overwrote the first delivery; either way the count still reached the total, the merge stopped waiting, and the conversion succeeded with layers silently missing from the output. Check each delivery: an index outside the image fails the merge with `Error::LayerIndexOutOfRange`, and a second delivery of the same layer with the new `Error::DuplicateLayer`. `StreamingPacker` already rejected out-of-range indices itself, but neither it nor the streaming conversion functions caught duplicates. Signed-off-by: Steven Noonan <steven@edera.dev>
The streaming conversion functions took a `tokio::sync::mpsc::Receiver`, which tied callers to tokio's channel even when their layers came from some other source, and came as three per-format variants (`convert_mksquashfs_streaming`, `convert_tar_streaming` and `convert_dir_streaming`) of what `ImageSpec` already expresses. Replace them with a single `convert_streaming(layers, total_layers, spec)`, mirroring `convert(image_dir, spec)`, that takes any `Stream` of `Result<LayerBlob, E>` where `E` converts into a `BoxError`: a tokio channel through tokio-stream's `ReceiverStream`, a `futures` channel, or any stream combinator. An `Err` item still aborts the conversion, and comes back as the source of `Error::LayerSource`. That is the usual convention for consuming a fallible stream (`reqwest`'s `Body::wrap_stream` works the same way), and it saves a caller from pairing a "stream ended early" error with a failure it would otherwise have to keep track of separately. A stream that cannot fail is passed as `stream.map(Ok::<_, Infallible>)`. The `Stream` trait comes from futures-core, the small, dependency-free crate that tokio-stream, futures and other stream libraries build on. `StreamingPacker`'s push-based API is unchanged. BREAKING CHANGE: `convert_mksquashfs_streaming`, `convert_tar_streaming` and `convert_dir_streaming` are replaced by `convert_streaming`, which takes a `Stream` of layers and an `ImageSpec`; a tokio receiver can be wrapped in `tokio_stream::wrappers::ReceiverStream`. Signed-off-by: Steven Noonan <steven@edera.dev>
tycho
force-pushed
the
steven/feat/typed-api
branch
from
October 5, 2026 18:19
cc8ce8b to
68396fc
Compare
tycho
marked this pull request as ready for review
October 5, 2026 18:27
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Breaking API changes for 0.3.0, making the library easier to consume:
verifyandimage; the merge engine's internal modules are private.ocirender::Error, a non-exhaustive enum whose variants say what failed (the image layout, a particular layer, the caller's layer source, mksquashfs, the output, ...), instead ofanyhow::Error. Failures are now pinned on the right side of a merge: a corrupt layer is reported as that layer's error, and mksquashfs dying is reported with its stderr rather than as a broken pipe.convert_streamingtakes anyStreamof layers plus anImageSpec, replacing the threeconvert_*_streamingfunctions that took a tokio channel.Also fixes the streaming merge silently dropping layers delivered with an out-of-range or duplicate index.
Existing
StreamingPackercallers should mostly compile unchanged:notify_erroraccepts any error type,anyhow::Errorincluded, andocirender::Errorconverts intoanyhow::Errorwith?. Protect's OCI crate builds against this branch without changes.Fixes #15
Fixes #16