Iobase streaming support in FileBasedSink, exposed in TextIO#35253

Merged

shunping merged 144 commits intoapache:masterfrom

razvanculea:iobase_streaming_textio

Jul 30, 2025

Commits on Jun 10, 2025

fixes per https://github.com/apache/beam/pull/35137#pullrequestreview-2906007316
razvanculea
committed

Commits on Jun 16, 2025

keep in sync with refactor in https://github.com/apache/beam/pull/35137
razvanculea
committed

Commits on Jun 17, 2025

keep in sync with iobase changes in https://github.com/apache/beam/pull/35137
razvanculea
committed
Merge branch 'master' into iobase_streaming_textio
razvanculea
authored
[Website] add akvelon case study (#34943)

authored and
razvanculea
committed
added the rate test for GenerateSequence (#35108)

liferoad
authored and
razvanculea
committed
Re-enable logging after importing vllm. (#35103)

authored and
razvanculea
committed
Deprecate Java 8 (#35064)

Abacn
authored and
razvanculea
committed
Remove beam college banners (#35123)

damccorm
authored and
razvanculea
committed
feat: change text (#35130)

authored and
razvanculea
committed
Update Custom Remote Inference example to use RemoteModelHandler (#35066)

authored and
razvanculea
committed
Remove Java 8 container (#35125)

Abacn
authored and
razvanculea
committed
add extra_transforms block documentation to chain transform documentation (#35101)

derrickaw
authored and
razvanculea
committed
add note about testing (#35075)

derrickaw
authored and
razvanculea
committed
fix whitespaces/newlines
razvanculea
committed
fixes per https://github.com/apache/beam/pull/35137#pullrequestreview-2906007316
razvanculea
committed
implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink
razvanculea
committed
update changes.md
razvanculea
committed
fix formatting
razvanculea
committed
space formatting
razvanculea
committed
space nagging
razvanculea
committed
keep in sync with refactor in https://github.com/apache/beam/pull/35137
razvanculea
committed
keep in sync with iobase changes in https://github.com/apache/beam/pull/35137
razvanculea
committed
[Website] update akvelon case study: update text and fix landing page (#35133)

authored and
razvanculea
committed
Fix PostCommit Python Xlang IO Dataflow job (#35131)

Amar3tto
authored and
razvanculea
committed
Bump google.golang.org/grpc from 1.72.0 to 1.72.2 in /sdks (#35113)

dependabot[bot]
authored and
razvanculea
committed
Bump cloud.google.com/go/bigquery from 1.67.0 to 1.69.0 in /sdks (#35061)

dependabot[bot]
authored and
razvanculea
committed
Add known issues. (#35138)

authored and
razvanculea
committed
Bump @octokit/plugin-paginate-rest and @octokit/rest (#34167)

dependabot[bot]
authored and
razvanculea
committed
Explicitly handle singleton iterators instead of using helper and catching exceptions which may be from generating iterable (#35124)

scwhittle
authored and
razvanculea
committed
Build last snapshot against RC00 tag instead of release branch (#35142)

Abacn
authored and
razvanculea
committed
Bump nodemailer from 6.7.5 to 6.9.9 in /scripts/ci/issue-report (#35143)

dependabot[bot]
authored and
razvanculea
committed
Fix tests affected by Java 8 container turn down (#35145)

Abacn
authored and
razvanculea
committed
Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#35146)

dependabot[bot]
authored and
razvanculea
committed
fix jdbc transform validation (#35141)

stankiewicz
authored and
razvanculea
committed
Fix Java Example ARM PostCommit

Abacn
authored and
razvanculea
committed
fix: add missed word (#35163)

authored and
razvanculea
committed
Add postcommit yaml xlang workflow and split tests accordingly (#35119)

derrickaw
authored and
razvanculea
committed
Replace usages of deprecated pkg_resources package (#35153)

jrmccluskey
authored and
razvanculea
committed
Improve error message when accidentally using PBegin/Pipeline (#35156)

hjtran
authored and
razvanculea
committed
add friendly error message for when transform is applied to no output (#35160)

hjtran
authored and
razvanculea
committed
Add warning if temp location bucket has soft delete enabled for Go SD… (#34996)

TanuSharma2511
authored and
razvanculea
committed
Constrain DequeCoder type correctly, as it does not support nulls

kennknowles
authored and
razvanculea
committed
Do not overwrite class states if a cached dynamic class is returned in cloudpickle.load (#35063)

baeminbo
authored and
razvanculea
committed
Make SDK harness change effective on Iceberg Dataflow test (#35173)

Abacn
authored and
razvanculea
committed
Fix beam_PostCommit_Java_Examples_Dataflow_V2 (#35172)

Abacn
authored and
razvanculea
committed
[YAML]: Update postgres IT test and readme (#35169)

derrickaw
authored and
razvanculea
committed
Bump Java beam-master container (#35170)

Abacn
authored and
razvanculea
committed
Make WindowedValue a public interface

kennknowles
authored and
razvanculea
committed
Run integration tests for moving WindowedValue and making public

kennknowles
authored and
razvanculea
committed
Add timer tests to make sure event-time timer firing at the right time. (#35109)

shunping
authored and
razvanculea
committed
change to ubuntu-20.04 (#35182)

derrickaw
authored and
razvanculea
committed
Fix IntelliJ sync project failure due to circular Beam dependency (#35167)

Abacn
authored and
razvanculea
committed
Update workflows categories (#35162)

Amar3tto
authored and
razvanculea
committed
Add cloudpickle coder. (#35166)

claudevdm
authored and
razvanculea
committed
Move examples from sql package (#35183)

Amar3tto
authored and
razvanculea
committed
Fix the beam interactive install problem when on Google Colab (#35148)

Chenzo1001
authored and
razvanculea
committed
Bump github.com/docker/docker in /sdks (#35112)

dependabot[bot]
authored and
razvanculea
committed
Bump github.com/nats-io/nats-server/v2 from 2.11.3 to 2.11.4 in /sdks (#35161)

dependabot[bot]
authored and
razvanculea
committed
Touch JPMS test trigger file

kennknowles
authored and
razvanculea
committed
Use staged SDK harness & Dataflow worker jar in JPMS tests

kennknowles
authored and
razvanculea
committed
Bump cloud.google.com/go/storage from 1.52.0 to 1.55.0 in /sdks (#35114)

dependabot[bot]
authored and
razvanculea
committed
Bump github.com/nats-io/nats.go from 1.42.0 to 1.43.0 in /sdks (#35147)

dependabot[bot]
authored and
razvanculea
committed
Move non-dill specific code out of dill_pickler.py (#35139)

kristynsmith
authored and
razvanculea
committed
Fix some lints introduced in a recent PR. (#35193)

shunping
authored and
razvanculea
committed
small filesystem fixes (#34956)

hjtran
authored and
razvanculea
committed
[YAML] Add a spec provider for transforms taking specifiable arguments (#35187)

shunping
authored and
razvanculea
committed
Touch trigger files to test WindowedValueReceiver in runners

kennknowles
authored and
razvanculea
committed
Introduce WindowedValue receivers and consolidate runner code to them

kennknowles
authored and
razvanculea
committed
Eliminate nullness errors from ByteBuddyDoFnInvokerFactory and DoFnOutputReceivers

kennknowles
authored and
razvanculea
committed
Fix null check when fetching driverJars from value provider

Abacn
authored and
razvanculea
committed
Fix PostCommit Python ValidatesRunner Samza / Spark jobs (#35210)

Amar3tto
authored and
razvanculea
committed
Update pypi documentation 30145 (#34329)

rakeshcusat
authored and
razvanculea
committed
Add more YAML examples involving Kafka and Iceberg (#35151)

charlespnh
authored and
razvanculea
committed
Evaluate namedTuples as equivalent to rows (#35218)

damccorm
authored and
razvanculea
committed
Add a new experiment flag to enable real-time clock as processing time. (#35202)

shunping
authored and
razvanculea
committed
Touch trigger files for lightweight runners

kennknowles
authored and
razvanculea
committed
Eliminate WindowedValue.getPane() in preparation for making it a user-facing interface

kennknowles
authored and
razvanculea
committed
Do not fail if there were failed deletes (#35222)

Amar3tto
authored and
razvanculea
committed
Force FnApiRunner in cases where prism can't handle use case (#35219)

damccorm
authored and
razvanculea
committed
Bump golang.org/x/net from 0.40.0 to 0.41.0 in /sdks (#35206)

dependabot[bot]
authored and
razvanculea
committed
Fix incorrect typehints generated by FlatMap with default identity function (#35164)

hjtran
authored and
razvanculea
committed
Parse values returned from Dataflow API to BoundedTrieData (#34738)

Abacn
authored and
razvanculea
committed
Remove breaking PDone change (#35224)

damccorm
authored and
razvanculea
committed
Generic Postgres + Cloudsql postgres embeddings. (#35215)

authored and
razvanculea
committed
Allow only one thread at a time to start the VLLM server. (#35234)

tvalentyn
authored and
razvanculea
committed
[IcebergIO] Create namespaces if needed (#35228)

ahmedabu98
authored and
razvanculea
committed
Support configuring flush_count and max_row_bytes of WriteToBigTable (#34761)

nguymin4
authored and
razvanculea
committed
Update CHANGES.md (#35242)

tvalentyn
authored and
razvanculea
committed
[Beam SQL] Implement Catalog and CatalogManager (#35223)

ahmedabu98
authored and
razvanculea
committed
[IcebergIO] Fix conversion logic for arrays of structs and maps of structs; fix output Schema resolution with column pruning (#35230)

ahmedabu98
authored and
razvanculea
committed
fix long_description when the md file cannot be found (#35246)

liferoad
authored and
razvanculea
committed
[IcebergIO] Create tables with a partition spec (#34966)

ahmedabu98
authored and
razvanculea
committed
Fix typo java-dependencies.md (#35251)

baeminbo
authored and
razvanculea
committed
Adding project and database support in write transform for firestoreIO (#35017)

TanuSharma2511
authored and
razvanculea
committed
Fix a logical type issue about JdbcDateType and JdbcTimeType (#35243)

shunping
authored and
razvanculea
committed
Remove internal code. (#35239)

authored and
razvanculea
committed
enable setting max_writer_per_bundle for avroIO and other IO (#35092)

TanuSharma2511
authored and
razvanculea
committed
[Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. (#35252)

scwhittle
authored and
razvanculea
committed
[IcebergIO] Integrate with Beam SQL (#34799)

ahmedabu98
authored and
razvanculea
committed
update changes (#35256)

ahmedabu98
authored and
razvanculea
committed
[YAML]: fix postcommit oracle bug and reorganize postcommit tests (#35191)

derrickaw
authored and
razvanculea
committed
update hadoop prefix (#35257)

ahmedabu98
authored and
razvanculea
committed
Set go version 1.24.4 (#35261)

Amar3tto
authored and
razvanculea
committed
upgrade org.apache.parquet:parquet-avro to to 1.15.2 (#35037)

authored and
razvanculea
committed
Bump google.golang.org/grpc from 1.72.2 to 1.73.0 in /sdks (#35236)

dependabot[bot]
authored and
razvanculea
committed
add more doc strings for integration tests (#35171)

derrickaw
authored and
razvanculea
committed
[IcebergIO] extend table partitioning to Beam SQL (#35268)

ahmedabu98
authored and
razvanculea
committed
Moving to 2.67.0-SNAPSHOT on master branch.

Amar3tto
authored and
razvanculea
committed
Update CHANGES.md to have fields for 2.67.0 release

Amar3tto
authored and
razvanculea
committed
add free disk space step (#35260)

derrickaw
authored and
razvanculea
committed
[yaml]: Fix post commit yaml once more (#35273)

derrickaw
authored and
razvanculea
committed
Polish anomaly detection notebook and get ready to be imported in public devsite. (#35278)

shunping
authored and
razvanculea
committed
Suppress Findbugs (#35274)

Abacn
authored and
razvanculea
committed
Support Java 17+ compiled Beam components for build, release, and xlang tests (#35232)

Abacn
authored and
razvanculea
committed
[IO] Update Debezium in DebeziumIO to 3.1.1 (#34763)

authored and
razvanculea
committed
Document BQ Storage API pipeline options (#35259)

VeronicaWasson
authored and
razvanculea
committed
Speed up StreamingDataflowWorkerTest by removing 10 second wait from shutdown path. (#35275)

scwhittle
authored and
razvanculea
committed
Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301)

dependabot[bot]
authored and
razvanculea
committed
Bump google.golang.org/api from 0.235.0 to 0.237.0 in /sdks (#35302)

dependabot[bot]
authored and
razvanculea
committed
Bump github.com/linkedin/goavro/v2 from 2.13.1 to 2.14.0 in /sdks (#35205)

dependabot[bot]
authored and
razvanculea
committed
Update spotbugs version, fix runner ubuntu version, fix found spotbugs issues (#35303)

Amar3tto
authored and
razvanculea
committed
feat: Add option to control resource cleanup failure for IT (#35287)

liferoad
authored and
razvanculea
committed
Revert "Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301)" (#35305)

damccorm
authored and
razvanculea
committed
Add PeriodicStream in the new time series folder. (#35300)

shunping
authored and
razvanculea
committed
try buildah to replace kaniko (#35289)

liferoad
authored and
razvanculea
committed
Adding error handler for SpannerReadSchemaTransformProvider and missi… (#35241)

TanuSharma2511
authored and
razvanculea
committed
requests vulnerability. (#35308)

claudevdm
authored and
razvanculea
committed
[IcebergIO] create custom java container image for tests (#35307)

ahmedabu98
authored and
razvanculea
committed
add streaming support to iobase (python) (#35137)
razvanculea
committed
Merge branch 'iobase_streaming_textio' of https://github.com/razvanculea/beam into iobase_streaming_textio
razvanculea
committed

Commits on Jul 30, 2025