Skip to content

[SPARK-14804][Graphx] Graph vertexRDD/EdgeRDD checkpoint results Clas… - #15447

Closed
apivovarov wants to merge 1 commit into
apache:masterfrom
apivovarov:14804
Closed

[SPARK-14804][Graphx] Graph vertexRDD/EdgeRDD checkpoint results Clas…#15447
apivovarov wants to merge 1 commit into
apache:masterfrom
apivovarov:14804

Conversation

@apivovarov

@apivovarovapivovarov commented Oct 12, 2016•
edited
Loading

Copy link
Copy Markdown
Contributor

EdgeRDD/VertexRDD wrap partitionsRDD
e.g. EdgeRDDImpl.checkpoint() calls partitionsRDD.checkpoint()
EdgeRDD/VertexRDD isCheckpointed() method should be implemented the same way - it should call partitionsRDD.isCheckpointed

Spark-shell test works:

import org.apache.spark.graphx._
val users = sc.parallelize(List((3L, "lucas"), (7L, "john"), (5L, "matt"), (2L, "kelly")))
val rel = sc.parallelize(List(Edge(3L, 7L, "collab"), Edge(5L, 3L, "advisor"), Edge(2L, 5L, "colleague"), Edge(5L, 7L, "pi")))
sc.setCheckpointDir("/tmp/check")
val g = Graph(users, rel)
g.checkpoint
g.vertices.collect
g.edges.collect
g.isCheckpointed // true
// try to collect EdgeRDD/VertexRDD again
g.vertices.collect // works
g.edges.collect // works

@apivovarov

apivovarov commented Oct 12, 2016

Copy link
Copy Markdown
ContributorAuthor

Related PRs
#15396
#12576

@SparkQA

SparkQA commented Oct 12, 2016

Copy link
Copy Markdown

Test build #3329 has finished for PR 15447 at commit b123b68.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@apivovarovapivovarov mentioned this pull request Oct 12, 2016
Closed
@maropumaropu mentioned this pull request Apr 23, 2017
Closed
maropu added a commit to maropu/spark that referenced this pull request Apr 23, 2017
@maropu
Closesapache#12835
559854f
Closesapache#11785Closesapache#13027Closesapache#13614Closesapache#13761Closesapache#15197Closesapache#14006Closesapache#12576Closesapache#15447Closesapache#13259Closesapache#15616Closesapache#14473Closesapache#16638Closesapache#16146Closesapache#17269Closesapache#17313Closesapache#17418Closesapache#17485Closesapache#17551Closesapache#17463Closesapache#17625Closesapache#10739Closesapache#15193Closesapache#15344Closesapache#14804Closesapache#16993Closesapache#17040Closesapache#15180Closesapache#17238
@felixcheung

felixcheung commented Apr 25, 2017

Copy link
Copy Markdown
Member

@apivovarov do we need this fix?

peter-toth pushed a commit to peter-toth/spark that referenced this pull request Oct 6, 2018
@maropuMingjie Tang
[BUILD] Close stale PRs
982bf88
This pr proposed to close stale PRs. Currently, we have 400+ open PRs and there are some stale PRs whose JIRA tickets have been already closed and whose JIRA tickets does not exist (also, they seem not to be minor issues).
// Open PRs whose JIRA tickets have been already closed
Closesapache#11785Closesapache#13027Closesapache#13614Closesapache#13761Closesapache#15197Closesapache#14006Closesapache#12576Closesapache#15447Closesapache#13259Closesapache#15616Closesapache#14473Closesapache#16638Closesapache#16146Closesapache#17269Closesapache#17313Closesapache#17418Closesapache#17485Closesapache#17551Closesapache#17463Closesapache#17625
// Open PRs whose JIRA tickets does not exist and they are not minor issues
Closesapache#10739Closesapache#15193Closesapache#15344Closesapache#14804Closesapache#16993Closesapache#17040Closesapache#15180Closesapache#17238
N/A
Author: Takeshi Yamamuro <yamamuro@apache.org>
Closesapache#17734 from maropu/resolved_pr.
Change-Id: Id2e590aa7283fe5ac01424d30a40df06da6098b5
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@maropu@zifeif2
[BUILD] Close stale PRs
46744e5
## What changes were proposed in this pull request?
This pr proposed to close stale PRs. Currently, we have 400+ open PRs and there are some stale PRs whose JIRA tickets have been already closed and whose JIRA tickets does not exist (also, they seem not to be minor issues).
// Open PRs whose JIRA tickets have been already closed
Closesapache#11785Closesapache#13027Closesapache#13614Closesapache#13761Closesapache#15197Closesapache#14006Closesapache#12576Closesapache#15447Closesapache#13259Closesapache#15616Closesapache#14473Closesapache#16638Closesapache#16146Closesapache#17269Closesapache#17313Closesapache#17418Closesapache#17485Closesapache#17551Closesapache#17463Closesapache#17625
// Open PRs whose JIRA tickets does not exist and they are not minor issues
Closesapache#10739Closesapache#15193Closesapache#15344Closesapache#14804Closesapache#16993Closesapache#17040Closesapache#15180Closesapache#17238
## How was this patch tested?
N/A
Author: Takeshi Yamamuro <yamamuro@apache.org>
Closesapache#17734 from maropu/resolved_pr.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@apivovarov@SparkQA@felixcheung