Skip to content

Move stagingLocation and tempLocation to GcsOptions - #31

Closed
peihe wants to merge 0 commit into
apache:masterfrom
peihe:temp-option
Closed

Move stagingLocation and tempLocation to GcsOptions#31
peihe wants to merge 0 commit into
apache:masterfrom
peihe:temp-option

Conversation

@peihe

@peihepeihe commented Mar 8, 2016

Copy link
Copy Markdown
Contributor

This will allow BigQueryOptions to extend GcsOptions, and BigQueryIO can access them through BigQueryOptions.

@peihe

peihe commented Mar 8, 2016

Copy link
Copy Markdown
ContributorAuthor

R: @lukecwik@dhalperi

@davorbonaci

davorbonaci commented Mar 9, 2016

Copy link
Copy Markdown
Member

Hm... Can you argue more why this is the right change?

GcsOptions should be about accessing GCS itself, but not necessarily job-wide settings like stagingLocation. Also, it is unclear that there's no a converse case -- where somebody could benefit from this actually being in DataflowPipelineOptions.

@peihe

peihe commented Mar 9, 2016

Copy link
Copy Markdown
ContributorAuthor

The reason is BigQueryIO needs a tempLocation option across runners.
The issue is the default value of tempLocation is coming from stagingLocation.
That is why I pull both of them to GcsOptions.

Another option is to pull both of them to a new PipelineOption.

@davorbonaci

davorbonaci commented Mar 9, 2016

Copy link
Copy Markdown
Member

Sounds like we might need a job-wide temporary location. I think this needs some discussion.

@dhalperi

dhalperi commented Mar 10, 2016

Copy link
Copy Markdown
Contributor

This one is tough -- the tempLocation here needs to be a GCS bucket -- BigQuery can only export/import to/from GCS. So this needs to be runner-independent (can't use DataflowPipelineOptions.getLempLocation()) but needs to be GCS/GCP/BigQuery specific.

Would be nice to be able to reuse the one from DataflowPipelineOptions if such exists.

@peihe
peiheforce-pushed the temp-option branch 2 times, most recently from f95b969 to a78a806CompareMarch 14, 2016 21:55
@peihepeihe closed this Mar 14, 2016
echauchot added a commit to echauchot/beam that referenced this pull request May 12, 2017
@echauchot
Fix query10: Add comment for strange groupByKey
1a8f83c
issue apache#31
cosmoskitten pushed a commit to cosmoskitten/beam that referenced this pull request Jun 16, 2017
@echauchot
Improve query5, query10 and query11
39f51ba
query5: Add comment on key lifting (issue apache#30)
query10: Add comment for strange groupByKey (issue apache#31)
query11: Replace Count.perKey by Count.perElement (issue apache#32)
asfgit pushed a commit that referenced this pull request Aug 23, 2017
@echauchot@iemejia
Improve query5, query10 and query11
7bfc982
query5: Add comment on key lifting (issue #30)
query10: Add comment for strange groupByKey (issue #31)
query11: Replace Count.perKey by Count.perElement (issue #32)
lukecwik referenced this pull request in lukecwik/incubator-beam Mar 22, 2018
@bsidhom
Merge pull request#31from bsidhom/data-channel-open
0ee6547
Don't close data channels from environment sessions
mareksimunek pushed a commit to mareksimunek/beam that referenced this pull request May 9, 2018
@vanekjar
apache#31[euphoria-core] Implementation of accumulator API + integra…
0effeed
…tion with Flink executor
mareksimunek pushed a commit to mareksimunek/beam that referenced this pull request May 9, 2018
@vanekjar
apache#31Refactored according to review feedback
a5c3433
mareksimunek pushed a commit to mareksimunek/beam that referenced this pull request May 9, 2018
@xitep
Merge pull requestapache#132from seznam/vanekjar/31/Accumulators
0289e01
apache#31 [euphoria-core] Implementation of accumulator API
dmvk pushed a commit to dmvk/beam that referenced this pull request May 15, 2018
@vanekjarDavid Moravek
apache#31[euphoria-core] Implementation of accumulator API + integra…
370c853
…tion with Flink executor
dmvk pushed a commit to dmvk/beam that referenced this pull request May 15, 2018
@vanekjarDavid Moravek
apache#31Refactored according to review feedback
7477a0f
tvalentyn pushed a commit to tvalentyn/beam that referenced this pull request May 15, 2018
@dhalperi
Closesapache#31
afccb07
mareksimunek referenced this pull request in seznam/beam Jul 9, 2018
@vanekjarmarek.simunek
#31[euphoria-core] Implementation of accumulator API + integration w…
ed2651d
…ith Flink executor
mareksimunek referenced this pull request in seznam/beam Jul 9, 2018
@vanekjarmarek.simunek
#31Refactored according to review feedback
b1f1f6c
dmvk referenced this pull request in seznam/beam Aug 17, 2018
@vanekjarDavid Moravek
#31[euphoria-core] Implementation of accumulator API + integration w…
9c38967
…ith Flink executor
dmvk referenced this pull request in seznam/beam Aug 17, 2018
@vanekjarDavid Moravek
#31Refactored according to review feedback
3899c5d
kennknowles pushed a commit that referenced this pull request Oct 16, 2018
@vanekjarDavid Moravek
#31[euphoria-core] Implementation of accumulator API + integration w…
d8c5ffa
…ith Flink executor
kennknowles pushed a commit that referenced this pull request Oct 16, 2018
@vanekjarDavid Moravek
#31Refactored according to review feedback
62696db
hengfengli referenced this pull request in hengfengli/beam Mar 21, 2022
@thiagotnunes
feat: metrics for read latency and logging (#31)
b9fbc97
* feat: add new data record read metrics
Adds the following metrics:
- Initial partition from created to scheduled state time (in ms)
- Data record commit timestamp to read time (in ms)
- Data record read time to emitted time (in ms)
* feat: refine logging
Adds the partition token to all possible logging. Add temporary metric
of data record count as well.
* chore: spotless apply
sjvanrossum pushed a commit to sjvanrossum/beam that referenced this pull request May 17, 2023
@laysakura
Merge pull requestapache#31from laysakura/merge/nivaldoh/pull/21
0b81aed
Merge nivaldoh#21
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@peihe@davorbonaci@dhalperi