Skip to content

[Spark load][Fe 6/6] Fe process etl and loading state job - #3717

Merged
morningman merged 3 commits into
apache:masterfrom
wyb:spark_load_fe_etl_loading
Jun 21, 2020
Merged

[Spark load][Fe 6/6] Fe process etl and loading state job#3717
morningman merged 3 commits into
apache:masterfrom
wyb:spark_load_fe_etl_loading

Conversation

@wyb

@wybwyb commented May 28, 2020

Copy link
Copy Markdown
Contributor

  1. Fe checks the status of etl job regularly
    1.1 If status is RUNNING, update etl job progress
    1.2 If status is CANCELLED, cancel load job
    1.3 If status is FINISHED, get the etl output file paths, update job state to LOADING and log job update info

  2. Fe sends PushTask to Be and commits transaction after all push tasks execute successfully

#3433

@wybwyb mentioned this pull request May 28, 2020
Closed
@imayimay added api-review Categorizes an issue or PR as actively needing an API review. area/load Issues or PRs related to all kinds of load kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. labels May 29, 2020
imay
imay reviewed May 30, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java Outdated
Comment threadfe/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java
Comment threadfe/src/main/java/org/apache/doris/load/loadv2/SparkLoadJob.java Outdated
@wybwyb changed the title [Spark load] Fe process etl and loading state job[Spark load][Fe 1/5] Fe process etl and loading state jobMay 30, 2020
@wybwyb changed the title [Spark load][Fe 1/5] Fe process etl and loading state job[Spark load][Fe 5/5] Fe process etl and loading state jobMay 30, 2020
@wybwyb changed the title [Spark load][Fe 5/5] Fe process etl and loading state job[Spark load][Fe 6/6] Fe process etl and loading state jobJun 10, 2020
@wyb
wybforce-pushed the spark_load_fe_etl_loading branch from d7c944a to b0aaf1eCompareJune 19, 2020 10:42
@wyb
wybforce-pushed the spark_load_fe_etl_loading branch from b0aaf1e to 1da4965CompareJune 19, 2020 11:07
morningman
morningman previously approved these changes Jun 19, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

morningman
morningman approved these changes Jun 20, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman

morningman commented Jun 20, 2020

Copy link
Copy Markdown
Contributor

The compile error of broker has been fix in #3913 .
So I just approve this PR.

@morningman
morningman merged commit a63fa88 into apache:masterJun 21, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@imayimayimay left review comments
@morningmanmorningmanmorningman approved these changes

Assignees

No one assigned

Labels

api-reviewCategorizes an issue or PR as actively needing an API review.area/loadIssues or PRs related to all kinds of loadkind/api-changeCategorizes issue or PR as related to adding, removing, or otherwise changing an APIkind/featureCategorizes issue or PR as related to a new feature.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wyb@morningman@imay