feat(azure_blob sink): add append blob support via blob_type option - #25627
feat(azure_blob sink): add append blob support via blob_type option#25627Danielku15 wants to merge 9 commits into
Conversation
github-actions
Bot
added
the
domain: sinks
github-actions
Bot
added
domain: external docs
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e5734e634c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…fault, ordering, docs)
|
Hi @Danielku15 , Thanks for this PR. I had to implement custom rsyslog extender for this, good to know that this is being worked up on. |
|
@harshavmb As soon the other PR of mine for azure blob storage tagging is merged, I will update this one and make it ready. Hence it strongly depends on how fast the vector folks are with their reviews. The code is mostly prepared locally already. 😉 |
pront
added
type: feature
Combines the newly-merged tags & metadata support (PR vectordotdev#25545) with the append-blob feature so that both features coexist: * Tags/metadata now apply to both `blob_type: block` and `blob_type: append`. Append blobs receive the tags/metadata on blob creation via `AppendBlobClientCreateOptions::{blob_tags_string, metadata}`. * Follows the reorganized layout from PR vectordotdev#25148: `AzureBlobRequest`, `AzureBlobResponse`, `AzureBlobMetadata`, `AzureBlobRetryLogic` and `build_client`/`build_healthcheck` live in `azure_blob::config`; `AzureBlobService`/`AzureBlobSink` live in `azure_blob::service`/`sink`. * `AzureBlobType` moves out of `azure_common::config` (blob-specific) into `azure_blob::config` alongside the other blob types. * Preserves append-only additions: type-aware defaults for `blob_time_format`/`blob_append_uuid`, concurrency=1 pin for `Append`, `batch.max_bytes` auto-default + startup validation, EAFP append flow with `BlockCountExceedsLimit`/`ContainerNotFound` diagnostics, and the full unit + integration test coverage plus a new `assert_append_blob_with_tags_and_metadata` cross-feature integration test.
…ned tag/append semantics
… to avoid hour-boundary flake
Aligns with the editorial style the reviewers applied to PR vectordotdev#25545: * drop the `**Header**:` bold section markers (no other Vector sink field uses that pattern), * remove the compressed-frames paragraph from the field description — the same information already lives on the `Append` enum-variant doc and therefore in the reference for the `append` value, * switch "Vector will fail to start" / "pins request concurrency to 1" to the passive phrasings used elsewhere ("rejected at startup", "defaults `request.concurrency` to `1`"), * deduplicate "4 MiB"/"4194304" mentions, and reword the batch bullet so the append blob key composition is explicit.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 894f43c22d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 894f43c22d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex fresh review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 894f43c22d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…, compression, retry docs) Rotate append blobs hourly by default (`%Y-%m-%dT%H`, the ISO 8601 reduced-precision hour form) instead of daily. Azure caps an append blob at 50,000 blocks and each flush consumes one, so daily rotation capped a partition near 2.3 MiB/s, after which every append failed with `BlockCountExceedsLimit` until the UTC date changed. Hourly rotation allows 50,000 flushes per hour, about 56 MiB/s at the 4 MiB batch limit. Reject `snappy` and `zlib` at startup when `blob_type` is `append`. Each batch is compressed on its own and appended as a separate block, so the blob only reads back if concatenated streams decode as a whole: `gzip` (multi-member) and `zstd` (multi-frame) qualify, Vector's raw Snappy blocks and bare zlib streams do not. zlib is why this is an error rather than a warning -- standard zlib decoders return only the first block and report success, so the loss is invisible to the consumer. Drop the claim that `request.retry_attempts = 0` yields at-most-once delivery. It only disables the sink-level retry; the batch is failed instead, and upstream retries or a resending source can still produce duplicates. Also extract `resolved_blob_naming()` so the `blob_type`-specific naming defaults are unit-testable without an Azurite round trip.
|
Remarks addressed |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d5c6934df
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…d blobs A block blob receives one self-contained payload per request, like the other object-store sinks, so framing the JSON codec as one array per blob is correct there. An append blob instead accumulates payloads into one growing blob -- the same shape the `file` sink writes -- and Azure concatenates them with nothing in between, so a second flush left the blob holding `[...][...]`, which no JSON or NDJSON parser accepts. Append mode now resolves codec defaults through `SinkType::StreamBased`, the same value the `file` sink uses, so `codec = "json"` without explicit framing writes newline-delimited JSON. Only the defaults are `blob_type`-aware: explicitly configured `framing` is passed through untouched, since delimiters and envelopes are the user's choice to make. The `blob_type` docs explain the seam behavior so that choice can be an informed one. Block mode is unchanged and keeps emitting one JSON array per blob.
|
Failed build of the playground seems unrelated to my changes: "The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.". I guess a GitHub outage is to blame here. |
3 participants
Summary
Adds
blob_type: appendsupport to theazure_blobsink, implementing #19397.The default behavior (
blob_type: block) is unchanged. Whenblob_type: appendis set, each flush appends to a stable-named Azure Append Blob rather than creating a new uniquely-named blob per batch — the natural model for continuous log streaming where you want a single growing file per time window.This branch is forward-merged onto master following the reorganization in #25148 and the tags & metadata feature in #25545. Blob tags and metadata now apply to both block and append blobs: for append mode they are set on the underlying
AppendBlobClientCreateOptionswhen the blob is first created (verified end-to-end by a new integration test).Key design decisions:
blob_time_formatdefaults to%Y-%m-%d(daily rotation) andblob_append_uuiddefaults tofalseforappend, matching the expected continuous-stream use case. Both can still be overridden.append_blockfirst (hot path = 1 API call for an existing blob), create on 404, retry. A 409 Conflict on create is swallowed — a concurrent writer created the blob first.BlockCountExceedsLimit(50k-block cap) andContainerNotFoundproduce actionable warnings.batch.max_bytesdefaults to the 4 MiBappend_blocklimit when unset (or when only implicitly at the 10 MB bulk default); explicit values above 4 MiB are rejected at startup.request.concurrencydefaults to1forappendmode (Azure orders appended blocks by receive-time, so parallel flushes could interleave). Users can still set it explicitly.request.retry_attempts = 0for at-most-once.gunzip,zstd -d) handle these correctly.Vector configuration
Minimal append blob configuration:
This produces a single blob per day (e.g.
app/2024-07-18.log) and appends each batch to it. Defaults areblob_time_format: "%Y-%m-%d",blob_append_uuid: false,batch.max_bytes: 4194304.Explicit batch size and custom rotation:
With tags and metadata (identical surface for block and append):
Important
SAS tokens need the
Add(orWrite) permission forblob_type: append, and theTagspermission whentagsis configured. See theconnection_stringfield docs for the full permission matrix.How did you test this PR?
cargo test --no-default-features --features sinks-azure_blob sinks::azure_blob): 36 pass. Tests added or updated by this branch:azure_blob_build_request_append_blob_defaults— daily time format, no UUID; bracketed with a clock window to avoid a UTC-midnight flake.azure_blob_build_request_append_blob_with_compression— gzip in append mode.azure_blob_append_blob_stable_name_without_uuid_and_time— stable key across flushes.azure_blob_append_blob_with_uuid_override_generates_unique_keys— explicit UUID override still supported.azure_blob_append_blob_custom_time_format_hourly_rotation— hourly rotation, bracketed with a clock window to avoid an hour-boundary flake.azure_blob_block_blob_request_carries_block_type— block dispatch still works.azure_blob_config_default_blob_type_is_block/_parse_blob_type_append— default + explicit parsing.azure_blob_append_blob_default_max_bytes_succeeds—blob_type: appendwith nobatch.max_bytesbuilds cleanly.azure_blob_append_blob_explicit_oversized_batch_fails_at_startup— > 4 MiB rejected with amax_bytes … exceedserror.azure_blob_append_blob_partial_batch_without_max_bytes_succeeds— regression for the Codex P2 remark: a[batch]table that sets onlytimeout_secsstill gets the 4 MiB append default rather than the 10 MB bulk default.azure_blob_append_blob_rejects_oversized_batch/_accepts_batch_at_limit— directlimit_max_bytesboundary checks.azure_blob_build_request_append_blob_with_tags_and_metadata— cross-feature: tags/metadata propagate through the append-blob request.cargo vdev int test azure, verified locally against Azurite 3.35.0):azure_blob_append_blob_reuses_same_blob[_with_oauth]— two batches land in one blob; content order preserved.azure_blob_append_blob_json_encoding[_with_oauth]— NDJSON content-type + line integrity across flushes.azure_blob_append_blob_default_daily_rotation[_with_oauth]— type-aware defaults resolved end-to-end; blob name contains today's%Y-%m-%d.azure_blob_append_blob_multiple_forced_flushes— smallbatch.max_bytesforces many blocks; all land in one blob.azure_blob_append_blob_with_tags_and_metadata[_with_oauth]— cross-feature: tags/metadata land on the created append blob (verified viaget_blob_tags/get_blob_metadata).Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.