Iobase streaming others - #35418

Merged

shunping merged 148 commits into

apache:masterfrom

razvanculea:iobase_streaming_others

Aug 7, 2025

Commits on Jun 3, 2025

Add support for streaming writes in IOBase (Python)
razvanculea
committed
1f52da7
Merge branch 'apache:master' into iobase-streaming
razvanculea
authored
d00dbc2
add triggering_frequency in iobase.Sink
razvanculea
committed
fe9eab2
Merge branch 'apache:master' into iobase-streaming
razvanculea
authored
e4ff191
Merge branch 'iobase-streaming' of https://github.com/razvanculea/beam into iobase-streaming
razvanculea
committed
db83273
fix whitespaces/newlines
razvanculea
committed
9cc97c6

Commits on Jun 10, 2025

fixes per https://github.com/apache/beam/pull/35137#pullrequestreview-2906007316
razvanculea
committed
6d1aa55

Commits on Jun 12, 2025

implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink
razvanculea
committed
4399005
update changes.md
razvanculea
committed
8de75c0
fix formatting
razvanculea
committed
b04e1ec
space formatting
razvanculea
committed
4c6dbc7
space nagging
razvanculea
committed
42e2cbc

Commits on Jun 16, 2025

keep in sync with refactor in https://github.com/apache/beam/pull/35137
razvanculea
committed
339d3cf

Commits on Jun 17, 2025

keep in sync with iobase changes in https://github.com/apache/beam/pull/35137
razvanculea
committed
c08c643
Merge branch 'master' into iobase_streaming_textio
razvanculea
authored
4695f1c
[Website] add akvelon case study (#34943)
authored and
razvanculea
committed
193dbe2
added the rate test for GenerateSequence (#35108)
liferoad
authored and
razvanculea
committed
f64732c
Re-enable logging after importing vllm. (#35103)
authored and
razvanculea
committed
96cf6d2
Deprecate Java 8 (#35064)
Abacn
authored and
razvanculea
committed
c8adaeb
Remove beam college banners (#35123)
damccorm
authored and
razvanculea
committed
7d30cc2
feat: change text (#35130)
authored and
razvanculea
committed
6dbc531
Update Custom Remote Inference example to use RemoteModelHandler (#35066)
authored and
razvanculea
committed
943ebf9
Remove Java 8 container (#35125)
Abacn
authored and
razvanculea
committed
3da6da8
add extra_transforms block documentation to chain transform documentation (#35101)
derrickaw
authored and
razvanculea
committed
e755d48
add note about testing (#35075)
derrickaw
authored and
razvanculea
committed
0dcc0d4
fix whitespaces/newlines
razvanculea
committed
82b134f
fixes per https://github.com/apache/beam/pull/35137#pullrequestreview-2906007316
razvanculea
committed
7089387
implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink
razvanculea
committed
e6fbcb0
update changes.md
razvanculea
committed
ebfd9f1
fix formatting
razvanculea
committed
66f67a1
space formatting
razvanculea
committed
87d0fac
space nagging
razvanculea
committed
6c2da61
keep in sync with refactor in https://github.com/apache/beam/pull/35137
razvanculea
committed
ad70d66
keep in sync with iobase changes in https://github.com/apache/beam/pull/35137
razvanculea
committed
f5796ca
[Website] update akvelon case study: update text and fix landing page (#35133)
authored and
razvanculea
committed
dfc6cfc
Fix PostCommit Python Xlang IO Dataflow job (#35131)
Amar3tto
authored and
razvanculea
committed
b888c89
Bump google.golang.org/grpc from 1.72.0 to 1.72.2 in /sdks (#35113)
dependabot[bot]
authored and
razvanculea
committed
0ef6c73
Bump cloud.google.com/go/bigquery from 1.67.0 to 1.69.0 in /sdks (#35061)
dependabot[bot]
authored and
razvanculea
committed
3cf1858
Add known issues. (#35138)
authored and
razvanculea
committed
b87416e
Bump @octokit/plugin-paginate-rest and @octokit/rest (#34167)
dependabot[bot]
authored and
razvanculea
committed
dc9c62b
Explicitly handle singleton iterators instead of using helper and catching exceptions which may be from generating iterable (#35124)
scwhittle
authored and
razvanculea
committed
d74e3bc
Build last snapshot against RC00 tag instead of release branch (#35142)
Abacn
authored and
razvanculea
committed
300e02b
Bump nodemailer from 6.7.5 to 6.9.9 in /scripts/ci/issue-report (#35143)
dependabot[bot]
authored and
razvanculea
committed
82765fe
Fix tests affected by Java 8 container turn down (#35145)
Abacn
authored and
razvanculea
committed
522b8bf
Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#35146)
dependabot[bot]
authored and
razvanculea
committed
2ef7eec
fix jdbc transform validation (#35141)
stankiewicz
authored and
razvanculea
committed
e5e02c0
Fix Java Example ARM PostCommit
Abacn
authored and
razvanculea
committed
f7f6608
fix: add missed word (#35163)
authored and
razvanculea
committed
e440709
Add postcommit yaml xlang workflow and split tests accordingly (#35119)
derrickaw
authored and
razvanculea
committed
3619ca2
Replace usages of deprecated pkg_resources package (#35153)
jrmccluskey
authored and
razvanculea
committed
f6faa68
Improve error message when accidentally using PBegin/Pipeline (#35156)
hjtran
authored and
razvanculea
committed
d083f15
add friendly error message for when transform is applied to no output (#35160)
hjtran
authored and
razvanculea
committed
9f485b8
Add warning if temp location bucket has soft delete enabled for Go SD… (#34996)
TanuSharma2511
authored and
razvanculea
committed
e283b4b
Constrain DequeCoder type correctly, as it does not support nulls
kennknowles
authored and
razvanculea
committed
96cc9c9
Do not overwrite class states if a cached dynamic class is returned in cloudpickle.load (#35063)
baeminbo
authored and
razvanculea
committed
e2a63af
Make SDK harness change effective on Iceberg Dataflow test (#35173)
Abacn
authored and
razvanculea
committed
0dcc4fc
Fix beam_PostCommit_Java_Examples_Dataflow_V2 (#35172)
Abacn
authored and
razvanculea
committed
a926d97
[YAML]: Update postgres IT test and readme (#35169)
derrickaw
authored and
razvanculea
committed
9e9f102
Bump Java beam-master container (#35170)
Abacn
authored and
razvanculea
committed
a9e11d7
Make WindowedValue a public interface
kennknowles
authored and
razvanculea
committed
d11ff01
Run integration tests for moving WindowedValue and making public
kennknowles
authored and
razvanculea
committed
6f424bf
Add timer tests to make sure event-time timer firing at the right time. (#35109)
shunping
authored and
razvanculea
committed
85d6de3
change to ubuntu-20.04 (#35182)
derrickaw
authored and
razvanculea
committed
d8fa868
Fix IntelliJ sync project failure due to circular Beam dependency (#35167)
Abacn
authored and
razvanculea
committed
c9bcec8
Update workflows categories (#35162)
Amar3tto
authored and
razvanculea
committed
d14eb7b
Add cloudpickle coder. (#35166)
claudevdm
authored and
razvanculea
committed
918472f
Move examples from sql package (#35183)
Amar3tto
authored and
razvanculea
committed
94b9032
Fix the beam interactive install problem when on Google Colab (#35148)
Chenzo1001
authored and
razvanculea
committed
344a59b
Bump github.com/docker/docker in /sdks (#35112)
dependabot[bot]
authored and
razvanculea
committed
6ffe4cf
Bump github.com/nats-io/nats-server/v2 from 2.11.3 to 2.11.4 in /sdks (#35161)
dependabot[bot]
authored and
razvanculea
committed
d5353e8
Touch JPMS test trigger file
kennknowles
authored and
razvanculea
committed
67cdf59
Use staged SDK harness & Dataflow worker jar in JPMS tests
kennknowles
authored and
razvanculea
committed
1d17368
Bump cloud.google.com/go/storage from 1.52.0 to 1.55.0 in /sdks (#35114)
dependabot[bot]
authored and
razvanculea
committed
1bfabc7
Bump github.com/nats-io/nats.go from 1.42.0 to 1.43.0 in /sdks (#35147)
dependabot[bot]
authored and
razvanculea
committed
5bfa884
Move non-dill specific code out of dill_pickler.py (#35139)
kristynsmith
authored and
razvanculea
committed
ed204f4
Fix some lints introduced in a recent PR. (#35193)
shunping
authored and
razvanculea
committed
1088a3a
small filesystem fixes (#34956)
hjtran
authored and
razvanculea
committed
379c15d
[YAML] Add a spec provider for transforms taking specifiable arguments (#35187)
shunping
authored and
razvanculea
committed
5a02cde
Touch trigger files to test WindowedValueReceiver in runners
kennknowles
authored and
razvanculea
committed
a2fe66b
Introduce WindowedValue receivers and consolidate runner code to them
kennknowles
authored and
razvanculea
committed
c4da6c6
Eliminate nullness errors from ByteBuddyDoFnInvokerFactory and DoFnOutputReceivers
kennknowles
authored and
razvanculea
committed
d08a063
Fix null check when fetching driverJars from value provider
Abacn
authored and
razvanculea
committed
e8f6f05
Fix PostCommit Python ValidatesRunner Samza / Spark jobs (#35210)
Amar3tto
authored and
razvanculea
committed
ae65ea9
Update pypi documentation 30145 (#34329)
rakeshcusat
authored and
razvanculea
committed
d0c3af6
Add more YAML examples involving Kafka and Iceberg (#35151)
charlespnh
authored and
razvanculea
committed
a1381ad
Evaluate namedTuples as equivalent to rows (#35218)
damccorm
authored and
razvanculea
committed
25ebd66
Add a new experiment flag to enable real-time clock as processing time. (#35202)
shunping
authored and
razvanculea
committed
90f328c
Touch trigger files for lightweight runners
kennknowles
authored and
razvanculea
committed
1beaaac
Eliminate WindowedValue.getPane() in preparation for making it a user-facing interface
kennknowles
authored and
razvanculea
committed
ab9e2af
Do not fail if there were failed deletes (#35222)
Amar3tto
authored and
razvanculea
committed
6aa4fa3
Force FnApiRunner in cases where prism can't handle use case (#35219)
damccorm
authored and
razvanculea
committed
d80cd9a
Bump golang.org/x/net from 0.40.0 to 0.41.0 in /sdks (#35206)
dependabot[bot]
authored and
razvanculea
committed
5fc0402
Fix incorrect typehints generated by FlatMap with default identity function (#35164)
hjtran
authored and
razvanculea
committed
67b9c6e
Parse values returned from Dataflow API to BoundedTrieData (#34738)
Abacn
authored and
razvanculea
committed
f3a4384
Remove breaking PDone change (#35224)
damccorm
authored and
razvanculea
committed
f69b82d
Generic Postgres + Cloudsql postgres embeddings. (#35215)
authored and
razvanculea
committed
63aedcf
Allow only one thread at a time to start the VLLM server. (#35234)
tvalentyn
authored and
razvanculea
committed
9776f5d
[IcebergIO] Create namespaces if needed (#35228)
ahmedabu98
authored and
razvanculea
committed
8282e0c
Support configuring flush_count and max_row_bytes of WriteToBigTable (#34761)
nguymin4
authored and
razvanculea
committed
8d17c6a
Update CHANGES.md (#35242)
tvalentyn
authored and
razvanculea
committed
e8fe493
[Beam SQL] Implement Catalog and CatalogManager (#35223)
ahmedabu98
authored and
razvanculea
committed
88a6270
[IcebergIO] Fix conversion logic for arrays of structs and maps of structs; fix output Schema resolution with column pruning (#35230)
ahmedabu98
authored and
razvanculea
committed
d51b27d
fix long_description when the md file cannot be found (#35246)
liferoad
authored and
razvanculea
committed
b635698
[IcebergIO] Create tables with a partition spec (#34966)
ahmedabu98
authored and
razvanculea
committed
d435782
Fix typo java-dependencies.md (#35251)
baeminbo
authored and
razvanculea
committed
a4b3a56
Adding project and database support in write transform for firestoreIO (#35017)
TanuSharma2511
authored and
razvanculea
committed
2ec2fb8
Fix a logical type issue about JdbcDateType and JdbcTimeType (#35243)
shunping
authored and
razvanculea
committed
5f46c87
Remove internal code. (#35239)
authored and
razvanculea
committed
5e95675
enable setting max_writer_per_bundle for avroIO and other IO (#35092)
TanuSharma2511
authored and
razvanculea
committed
aff00db
[Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. (#35252)
scwhittle
authored and
razvanculea
committed
5b79048
[IcebergIO] Integrate with Beam SQL (#34799)
ahmedabu98
authored and
razvanculea
committed
7bd7f35
update changes (#35256)
ahmedabu98
authored and
razvanculea
committed
d6b3182
[YAML]: fix postcommit oracle bug and reorganize postcommit tests (#35191)
derrickaw
authored and
razvanculea
committed
6a12c4e
update hadoop prefix (#35257)
ahmedabu98
authored and
razvanculea
committed
53feaf8
Set go version 1.24.4 (#35261)
Amar3tto
authored and
razvanculea
committed
ac15c12
upgrade org.apache.parquet:parquet-avro to to 1.15.2 (#35037)
authored and
razvanculea
committed
60d4854
Bump google.golang.org/grpc from 1.72.2 to 1.73.0 in /sdks (#35236)
dependabot[bot]
authored and
razvanculea
committed
1fe3aee
add more doc strings for integration tests (#35171)
derrickaw
authored and
razvanculea
committed
22a4799
[IcebergIO] extend table partitioning to Beam SQL (#35268)
ahmedabu98
authored and
razvanculea
committed
bfc2244
Moving to 2.67.0-SNAPSHOT on master branch.
Amar3tto
authored and
razvanculea
committed
59f8a84
Update CHANGES.md to have fields for 2.67.0 release
Amar3tto
authored and
razvanculea
committed
9715848
add free disk space step (#35260)
derrickaw
authored and
razvanculea
committed
b9e6a48
[yaml]: Fix post commit yaml once more (#35273)
derrickaw
authored and
razvanculea
committed
f8bc1b7
Polish anomaly detection notebook and get ready to be imported in public devsite. (#35278)
shunping
authored and
razvanculea
committed
3fc5653
Suppress Findbugs (#35274)
Abacn
authored and
razvanculea
committed
7dad0f0
Support Java 17+ compiled Beam components for build, release, and xlang tests (#35232)
Abacn
authored and
razvanculea
committed
422f214
[IO] Update Debezium in DebeziumIO to 3.1.1 (#34763)
authored and
razvanculea
committed
8c9c3e7
Document BQ Storage API pipeline options (#35259)
VeronicaWasson
authored and
razvanculea
committed
bfe510f
Speed up StreamingDataflowWorkerTest by removing 10 second wait from shutdown path. (#35275)
scwhittle
authored and
razvanculea
committed
827caf6
Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301)
dependabot[bot]
authored and
razvanculea
committed
d62a636
Bump google.golang.org/api from 0.235.0 to 0.237.0 in /sdks (#35302)
dependabot[bot]
authored and
razvanculea
committed
337156e
Bump github.com/linkedin/goavro/v2 from 2.13.1 to 2.14.0 in /sdks (#35205)
dependabot[bot]
authored and
razvanculea
committed
86a0947
Update spotbugs version, fix runner ubuntu version, fix found spotbugs issues (#35303)
Amar3tto
authored and
razvanculea
committed
e3062b0
feat: Add option to control resource cleanup failure for IT (#35287)
liferoad
authored and
razvanculea
committed
8e13061
Revert "Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301)" (#35305)
damccorm
authored and
razvanculea
committed
efb5831
Add PeriodicStream in the new time series folder. (#35300)
shunping
authored and
razvanculea
committed
f840882
try buildah to replace kaniko (#35289)
liferoad
authored and
razvanculea
committed
19b4de1
Adding error handler for SpannerReadSchemaTransformProvider and missi… (#35241)
TanuSharma2511
authored and
razvanculea
committed
cb80c16
requests vulnerability. (#35308)
claudevdm
authored and
razvanculea
committed
029b886
[IcebergIO] create custom java container image for tests (#35307)
ahmedabu98
authored and
razvanculea
committed
f8cb258
add streaming support to iobase (python) (#35137)
razvanculea
committed
738abd5
Merge branch 'iobase_streaming_textio' of https://github.com/razvanculea/beam into iobase_streaming_textio
razvanculea
committed
4120eab

Commits on Jun 24, 2025

add streaming to AvroIO, ParquetIO, TFRecordsIO
razvanculea
committed
fa8e102
reformat
razvanculea
committed
8d063fe
typo and spaces
razvanculea
committed
d9bb9e0

Commits on Jul 31, 2025

Merge branch 'master' into iobase_streaming_others
razvanculea
authored
77f955d
carry on the refactor from #35253
razvanculea
committed
dd013d1

Commits on Aug 6, 2025

Merge branch 'master' into iobase_streaming_others
razvanculea
committed
6680874