Skip to content

[Spark Core][MINOR] fix "default partitioner cannot partition array keys" error message in PairRDDfunctions - #15045

Closed
WeichenXu123 wants to merge 4 commits into
apache:masterfrom
WeichenXu123:fix_partitionBy_error_message
Closed

[Spark Core][MINOR] fix "default partitioner cannot partition array keys" error message in PairRDDfunctions#15045
WeichenXu123 wants to merge 4 commits into
apache:masterfrom
WeichenXu123:fix_partitionBy_error_message

Conversation

@WeichenXu123

@WeichenXu123WeichenXu123 commented Sep 10, 2016•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

In order to avoid confusing user,
error message in PairRDDfunctions
Default partitioner cannot partition array keys.
is updated,
the one in partitionBy is replaced with
Specified partitioner cannot partition array keys.
other is replaced with
Specified or default partitioner cannot partition array keys.

How was this patch tested?

N/A

@srowen

srowen commented Sep 10, 2016

Copy link
Copy Markdown
Member

There are 5 instances of this check in the file -- they should all be handled the same way. I'm not sure this is accurate either because some code paths lead to these methods when HashPartitioner is used as a default. Just say that HashPartitioner can't be used? refactor one check method for this?

@WeichenXu123

WeichenXu123 commented Sep 10, 2016

Copy link
Copy Markdown
ContributorAuthor

oh, there are 5 similar messages..
I check the others, the others may be set the default one,
so I update their message as "Specified or default partitioner..."
but the one in partitionBy must be set by user, can't use default one,
because the partitionBy API is let user specify how the RDD to be partitioned.
other cases, the partitioner parameter is optional and if user not specify it,
it will be set into the default HashPartitioner.
Now I update the code.
thanks!

@WeichenXu123WeichenXu123 changed the title [Spark Core][MINOR] fix partitionBy error message[Spark Core][MINOR] fix "default partitioner cannot partition array keys" error message in PairRDDfunctionsSep 10, 2016
@SparkQA

SparkQA commented Sep 10, 2016

Copy link
Copy Markdown

Test build #65206 has finished for PR 15045 at commit d423b41.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Sep 10, 2016

Copy link
Copy Markdown

Test build #65207 has finished for PR 15045 at commit 6520854.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen

srowen commented Sep 10, 2016

Copy link
Copy Markdown
Member

Why bother saying 'specified' or 'default' at all though? it's probably even more informative to state that HashPartitioner doesn't work, no matter what the source. If the user specified HashPartitioner, that's clear. If they didn't, they'll still recognize that the other half of the message is relevant: some thing doesn't like their array keys.

srowen
srowen reviewed Sep 11, 2016
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala Outdated
def partitionBy(partitioner: Partitioner): RDD[(K, V)] = self.withScope {
if (keyClass.isArray && partitioner.isInstanceOf[HashPartitioner]) {
throw new SparkException("Default partitioner cannot partition array keys.")
throw new SparkException("Specified partitioner cannot partition array keys.")

@srowensrowenSep 11, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even this method is called by other Spark code with HashPartitioner, which might lead to this error telling the user the "specified" partitioner doesn't work when the user code didn't specify a partitioner.

@SparkQA

SparkQA commented Sep 11, 2016

Copy link
Copy Markdown

Test build #65229 has finished for PR 15045 at commit f53ad51.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Sep 11, 2016

Copy link
Copy Markdown

Test build #65227 has finished for PR 15045 at commit 25f1f8c.

  • This patch fails from timeout after a configured wait of 250m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WeichenXu123

WeichenXu123 commented Sep 11, 2016

Copy link
Copy Markdown
ContributorAuthor

jenkins test please

@WeichenXu123

WeichenXu123 commented Sep 12, 2016

Copy link
Copy Markdown
ContributorAuthor

Jenkins, test this please.

@SparkQA

SparkQA commented Sep 12, 2016

Copy link
Copy Markdown

Test build #65236 has finished for PR 15045 at commit f53ad51.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen

srowen commented Sep 12, 2016

Copy link
Copy Markdown
Member

OK, can't hurt to be clear and specific about this. Merged to master

wgtmac pushed a commit to wgtmac/spark that referenced this pull request Sep 19, 2016
@WeichenXu123@wgtmac
[SPARK CORE][MINOR] fix "default partitioner cannot partition array k…
7037d03
…eys" error message in PairRDDfunctions
## What changes were proposed in this pull request?
In order to avoid confusing user,
error message in `PairRDDfunctions`
`Default partitioner cannot partition array keys.`
is updated,
the one in `partitionBy` is replaced with
`Specified partitioner cannot partition array keys.`
other is replaced with
`Specified or default partitioner cannot partition array keys.`
## How was this patch tested?
N/A
Author: WeichenXu <WeichenXu123@outlook.com>
Closesapache#15045 from WeichenXu123/fix_partitionBy_error_message.
@WeichenXu123
WeichenXu123 deleted the fix_partitionBy_error_message branch April 24, 2019 21:19
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@WeichenXu123@srowen@SparkQA