Skip to content

[SPARK-21045][PYTHON] Allow non-ascii string as an exception message from python execution in Python 2 - #25847

Closed
advancedxy wants to merge 7 commits into
apache:masterfrom
advancedxy:python_exception_19926_and_21045
Closed

[SPARK-21045][PYTHON] Allow non-ascii string as an exception message from python execution in Python 2#25847
advancedxy wants to merge 7 commits into
apache:masterfrom
advancedxy:python_exception_19926_and_21045

Conversation

@advancedxy

@advancedxyadvancedxy commented Sep 19, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR allows non-ascii string as an exception message in Python 2 by explicitly en/decoding in case of str in Python 2.

Why are the changes needed?

Previously PySpark will hang when the UnicodeDecodeError occurs and the real exception cannot be passed to the JVM side.

See the reproducer as below:

deff():
raiseException("中")
spark=SparkSession.builder.master('local').getOrCreate()
spark.sparkContext.parallelize([1]).map(lambdax: f()).count()

Does this PR introduce any user-facing change?

User may not observe hanging for the similar cases.

How was this patch tested?

Added a new test and manually checking.

This pr is based on #18324, credits should also go to @dataknocker.
To make lint-python happy for python3, it also includes a followup fix for #25814

@advancedxy

advancedxy commented Sep 19, 2019

Copy link
Copy Markdown
ContributorAuthor

cc @HyukjinKwon, @ueshin and @cloud-fan

HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/worker.py Outdated
HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/testing/utils.py Outdated
HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/tests/test_worker.py Outdated
self.assertIsInstance(t.exception, Py4JJavaError)
if sys.version_info.major < 3:
# we have to use unicode here to avoid UnicodeDecodeError
self.assertRegexpMatches(unicode(t.exception).encode("utf-8"), "exception with 中")

@HyukjinKwonHyukjinKwonSep 19, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, str against Py4j exception doesn't properly handle non-ascii codes (py4j/py4j#308)

HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/tests/test_worker.py
HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/tests/test_worker.py Outdated
@HyukjinKwon

HyukjinKwon commented Sep 19, 2019

Copy link
Copy Markdown
Member

ok to test

HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/worker.py Outdated
except Exception:
try:
exc_info = traceback.format_exc()
if sys.version_info.major < 3:

@HyukjinKwonHyukjinKwonSep 19, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, let's drop this right after we drop Python 2, which I will do right after Spark 3.

HyukjinKwon
HyukjinKwon approved these changes Sep 19, 2019
View reviewed changes

@HyukjinKwonHyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good otherwise.

@HyukjinKwonHyukjinKwon mentioned this pull request Sep 19, 2019
Closed
@HyukjinKwonHyukjinKwon changed the title [SPARK-21045][PYSPARK] Defensive check for exception info thrown by user[SPARK-21045][PYTHON] Allow non-ascii string as an exception message in Python 2Sep 19, 2019
@SparkQA

SparkQA commented Sep 19, 2019

Copy link
Copy Markdown

Test build #110987 has finished for PR 25847 at commit 90559c0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen
srowen reviewed Sep 19, 2019
View reviewed changes

@srowensrowen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, when does an exception have non-ASCII chars? when it reports a table name or input value from the user app?

HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/testing/utils.py Outdated
@HyukjinKwon

HyukjinKwon commented Sep 19, 2019

Copy link
Copy Markdown
Member

@srowen, for instance, users could manually throw an exception with python native function execution like udf or rdd.

@SparkQA

SparkQA commented Sep 19, 2019

Copy link
Copy Markdown

Test build #110997 has finished for PR 25847 at commit fb72447.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Sep 19, 2019

Copy link
Copy Markdown

Test build #111002 has finished for PR 25847 at commit ff7f248.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

HyukjinKwon
HyukjinKwon reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/worker.py Outdated
@HyukjinKwonHyukjinKwon changed the title [SPARK-21045][PYTHON] Allow non-ascii string as an exception message in Python 2[SPARK-21045][PYTHON] Allow non-ascii string as an exception message from python execution in Python 2Sep 19, 2019
@advancedxyadvancedxy mentioned this pull request Sep 19, 2019
Closed
advancedxy
advancedxy commented Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/worker.py Outdated

if sys.version >= '3':
basestring = str
unicode = str

@advancedxyadvancedxySep 19, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary, see #25814 (comment)

@HyukjinKwonHyukjinKwonSep 20, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I think we don't need the comditionat 603 line

@SparkQA

SparkQA commented Sep 19, 2019

Copy link
Copy Markdown

Test build #111007 has finished for PR 25847 at commit 0652966.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Sep 19, 2019

Copy link
Copy Markdown

Test build #111009 has finished for PR 25847 at commit ffb4d29.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

viirya
viirya reviewed Sep 19, 2019
View reviewed changes
Comment threadpython/pyspark/tests/test_worker.py
viirya
viirya reviewed Sep 19, 2019
View reviewed changes

@viiryaviirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good except for a question.

HyukjinKwon
HyukjinKwon reviewed Sep 20, 2019
View reviewed changes
Comment threadpython/pyspark/worker.py Outdated
@SparkQA

SparkQA commented Sep 20, 2019

Copy link
Copy Markdown

Test build #111077 has finished for PR 25847 at commit d6ec7ae.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Sep 20, 2019

Copy link
Copy Markdown
Member

Merged to master.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowensrowensrowen left review comments
@HyukjinKwonHyukjinKwonHyukjinKwon approved these changes
@viiryaviiryaviirya approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@advancedxy@HyukjinKwon@SparkQA@viirya@srowen@dongjoon-hyun