[SPARK-25108][SQL] Fix the show method to display the wide character alignment problem by xuejianbest · Pull Request #22048 · apache/spark

xuejianbest · 2018-08-09T03:31:58Z

This is not a perfect solution. It is designed to minimize complexity on the basis of solving problems.

It is effective for English, Chinese characters, Japanese, Korean and so on.

before:+---+---------------------------+-------------+|id |中国 |s2 |+---+---------------------------+-------------+|1|ab |[a] ||2|null|[中国, abc] ||3|ab1 |[hello world]||4|か行 きゃ(kya) きゅ(kyu) きょ(kyo) |[“中国] ||5|中国（你好）a |[“中（国）, 312] ||6|中国山(东)服务区 |[“中(国）] ||7|中国山东服务区 |[中(国)] ||8||[中国] |+---+---------------------------+-------------+
after:+---+-----------------------------------+----------------+|id |中国 |s2 |+---+-----------------------------------+----------------+|1|ab |[a] ||2|null|[中国, abc] ||3|ab1 |[hello world] ||4|か行 きゃ(kya) きゅ(kyu) きょ(kyo) |[“中国] ||5|中国（你好）a |[“中（国）, 312]||6|中国山(东)服务区 |[“中(国）] ||7|中国山东服务区 |[中(国)] ||8||[中国] |+---+-----------------------------------+----------------+

What changes were proposed in this pull request?

When there are wide characters such as Chinese characters or Japanese characters in the data, the show method has a alignment problem.
Try to fix this problem.

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

Please review http://spark.apache.org/contributing.html before opening a pull request.

before: +---+---------------------------+-------------+ |id |中国 |s2 | +---+---------------------------+-------------+ |1 |ab |[a] | |2 |null |[中国, abc] | |3 |ab1 |[hello world]| |4 |か行きゃ(kya) きゅ(kyu) きょ(kyo) |[“中国] | |5 |中国（你好）a |[“中（国）, 312] | |6 |中国山(东)服务区 |[“中(国）] | |7 |中国山东服务区 |[中(国)] | |8 | |[中国] | +---+---------------------------+-------------+ after: +---+--------------------------+----------------+ | id| 中国| s2| +---+--------------------------+----------------+ | 1| ab| [a]| | 2| null| [中国, abc]| | 3| ab1| [hello world]| | 4|か行きゃ(kya) きゅ(kyu...| [“中国]| | 5| 中国（你好）a|[“中（国）, 312]| | 6| 中国山(东)服务区| [“中(国）]| | 7| 中国山东服务区| [中(国)]| | 8| | [中国]| +---+--------------------------+----------------+

kiszk · 2018-08-09T06:16:35Z

Since this change does not look minor, could you please create a JIRA entry?

kiszk · 2018-08-09T06:38:45Z

I created a PR to solve the same problem.
Can this PR handle East Asian Width correctly?

xuejianbest · 2018-08-10T09:33:01Z

After testing, it is found that regular expressions are changed to the following.
val regex = """[^\x00-\u2e39]""".r

@kiszk
I solve this problem by regular expressions. The complexity and display effect are weighed.

kiszk · 2018-08-13T17:48:47Z

@xuejianbest Could you please create a JIRA entry and add test cases to this PR?

xuejianbest · 2018-08-14T02:27:33Z

Below is the display effect I tested on Xshell5.
https://issues.apache.org/jira/secure/attachment/12935462/show.bmp

xuejianbest · 2018-08-14T02:30:01Z

kiszk · 2018-08-17T00:47:21Z

Thank you for creating a JIRA entry and for putting the result. The test case is not available yet.

srowen · 2018-08-19T15:03:37Z

 // We set a minimum column width at '3'
 val minimumColWidth = 3

+ val regex = """[^\x00-\u2e39]""".r


This could use a comment and slightly better name for the variable? I also wonder if a regex is a little bit slow for scanning every character.
However it's not clear this definition is accurate enough. According to things like https://stackoverflow.com/questions/13505075/analyzing-full-width-or-half-width-character-in-java we're really looking for the concept of "fullwidth" East Asian characters. The answer there provides a somewhat more precise definition, though others say you need something like icu4j for a proper definition.
Maybe at least adopt the alternative proposed in the SO answer? It's not necessary to be perfect here, as it's a cosmetic issue.

I agree with @srowen's comment. To make it clear, could you please add tests for more characters?

srowen · 2018-08-26T20:47:45Z

Ping @xuejianbest

Modify regular expressions to make them more precise. Modify variable names and add comment.

xuejianbest · 2018-08-28T09:16:03Z

I looked at all the 0x0000-0xFFFF characters (unicode) and showed them under Xshell, then found all the full width characters. Get the latest regular expression.
A new version has been submitted, and the variable name has been changed and comments added, please see below.
@srowen @kiszk

A new version is submitted, and the variable name is changed and an annotation is added. See below.

kiszk · 2018-08-28T09:45:44Z

@xuejianbest thank you for your update. Would it be possible to commit test cases, too?

srowen · 2018-08-28T16:31:32Z

 // We set a minimum column width at '3'
 val minimumColWidth = 3

+ //Regular expression matching full width characters


Very small nit that might fail scalastyle -- space after comment slashes

OK, thanks.

srowen · 2018-08-28T16:32:18Z

 val minimumColWidth = 3

+ //Regular expression matching full width characters
+ val fullWidthRegex = """[\u1100-\u115F\u2E80-\uA4CF\uAC00-\uD7A3\uF900-\uFAFF\uFE10-\uFE19\uFE30-\uFE6F\uFF00-\uFF60\uFFE0-\uFFE6]""".r


I wonder if detecting this using a regex for each character is slow? probably not meaningfully enough to care about, but I wonder.

I generated 1000 strings, each consisting of 1000 characters with a random unicode of 0x0000-0xFFFF. (a total of 1 million characters.)
Then use this regular expression to find the full width character of these strings.
I tested 100 rounds and then averaged.
It takes 49 milliseconds to complete matching all 1000 strings.

OK, not worth optimizing now.

srowen · 2018-08-28T16:32:52Z

 for (row <- rows) {
 for ((cell, i) <- row.zipWithIndex) {
- colWidths(i) = math.max(colWidths(i), cell.length)
+ colWidths(i) = math.max(colWidths(i), cell.length + fullWidthRegex.findAllIn(cell).size)


What about a utility method for the x.length + fullWidthRegex.findAllIn(x).size expression that recurs here?

I committed a new version. See if this is appropriate please ?
@srowen

Putting `fullWidthRegex` outside `stringHalfWidth` is to reduce the cost of compiling regular expressions. Since compilation is expensive, frequently used Regexes should be constructed once.

maropu · 2018-08-29T03:51:59Z

+ // The number of half width of a string
+ def stringHalfWidth = (str: String) => {
+ str.length + fullWidthRegex.findAllIn(str).size
+ }


better to move this method into util.Utils or something as a helper function?

better to add tests for the helper function, too.

Moved it into util.Utils.

maropu · 2018-08-29T03:52:56Z

 val minimumColWidth = 3

+ // Regular expression matching full width characters
+ val fullWidthRegex = """[\u1100-\u115F\u2E80-\uA4CF\uAC00-\uD7A3\uF900-\uFAFF\uFE10-\uFE19\uFE30-\uFE6F\uFF00-\uFF60\uFFE0-\uFFE6]""".r


This line goes over the limit, 100.

Move the stringHalfWidth method into util.Utils and add tests for it.

xuejianbest · 2018-08-29T09:38:01Z

I have moved the method of calculating the half width of a string into util.Utils object.
@maropu @srowen

srowen · 2018-08-29T14:25:33Z

+ * A full width character occupies two half widths
+ */
+ def stringHalfWidth(str: String): Int = {
+ if(str == null) 0 else str.length + fullWidthRegex.findAllIn(str).size


Nit: space before if

srowen · 2018-08-29T14:25:45Z

+ /**
+ * Regular expression matching full width characters
+ */
+ private lazy val fullWidthRegex = ("""[""" +


Doesn't need to be lazy

maropu · 2018-08-29T23:47:37Z

Also, can you add tests in DatasetSuite?

SparkQA · 2018-08-30T13:48:19Z

Test build #4305 has finished for PR 22048 at commit 3649de5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

xuejianbest · 2018-08-31T01:34:30Z

Why did it fail?
What else should I do?

xuejianbest · 2018-08-31T01:35:49Z

Please help me @srowen
I don't know where I need to make changes.

srowen · 2018-08-31T02:08:50Z

Have a look at the test results, which show the style problem it flags:
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4305/console

[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2803:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2804:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2805:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2806:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2807:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2808:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2809:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2810:4: nonascii.message

So it doesn't like non-ASCII chars after all! I wonder how they exist in the other parts of the source. I checked and other parts just disable the check for sections of code where it's appropriate to use unicode chars.

Just wrap these parts of the code in:

// scalastyle:off nonascii
...
// scalastyle:on nonascii

Sorry, didn't know this was needed.

maropu · 2018-08-31T02:09:05Z

style errors, we cannot include non-ascii characters in files;

Scalastyle checks failed at following occurrences:
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2803:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2804:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2805:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2806:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2807:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2808:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2809:4: nonascii.message
[error] /home/jenkins/workspace/NewSparkPullRequestBuilder@2/core/src/main/scala/org/apache/spark/util/Utils.scala:2810:4: nonascii.message
[error] Total time: 16 s, completed Aug 30, 2018 6:48:19 AM
[error] running /home/jenkins/workspace/NewSparkPullRequestBuilder@2/dev/lint-scala ; received return code 1
Attempting to post to Github...
> Post successful.

maropu · 2018-08-31T02:10:08Z

+1 for adding scalastyle:off nonascii in that part.

To solve style errors because of include non-ascii characters in files. Disable the check for sections of code where it's appropriate to use unicode chars.

xuejianbest · 2018-08-31T03:01:25Z

I see. A new commit has been done.
Thinks.

kiszk · 2018-08-31T05:00:36Z

retest this please

srowen · 2018-08-31T13:51:16Z

Hm! so the warning triggers on unicode escape sequences? That seems like a bug in the style checker. Oh well, yes keep them. I'll retest ...

SparkQA · 2018-08-31T18:29:41Z

Test build #4329 has finished for PR 22048 at commit 45ac272.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-08-31T19:29:24Z

LGTM, then let me ask @gatorsmile

gatorsmile · 2018-08-31T22:46:49Z

+ """\uFE10-\uFE19""" +
+ """\uFE30-\uFE6F""" +
+ """\uFF00-\uFF60""" +
+ """\uFFE0-\uFFE6""" +


A general question.
How to get this Regex list? Any reference? It sounds like this should be a general problem
What is the performance impact?
Can you answer them and post them in the PR description?

How to get this Regex list? Any reference? It sounds like this should be a general problem
I looked at all the 0x0000-0xFFFF characters (unicode) and showed them under Xshell, then found all the full width characters. Get the regular expression.
What is the performance impact?
I generated 1000 strings, each consisting of 1000 characters with a random unicode of 0x0000-0xFFFF. (a total of 1 million characters.)
Then use this regular expression to find the full width character of these strings.
I tested 100 rounds and then averaged.
It takes 49 milliseconds to complete matching all 1000 strings.
@gatorsmile

I looked at all the 0x0000-0xFFFF characters (unicode) and showed them under Xshell, then found all the full width characters. Get the regular expression.
Can you describe them there and put a references to a public unicode document?
See the comment in UTF8String;
https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L65
It takes 49 milliseconds to complete matching all 1000 strings.
How about some additional overheads when calling showString as compared to showString w/o this patch?

Can you describe them there and put a references to a public unicode document?
This is a regular expression match using unicode, regardless of the specific encoding.
For example, the following string is encoded using gbk instead of utf8, and the match still works：
valbytes=Array[Byte](0xd6.toByte, 0xd0.toByte, 0xB9.toByte, 0xFA.toByte) vals1=newString(bytes, "gbk") println(s1) //中国 valfullWidthRegex= ("""["""+// scalastyle:off nonascii"""\u1100-\u115F"""+"""\u2E80-\uA4CF"""+"""\uAC00-\uD7A3"""+"""\uF900-\uFAFF"""+"""\uFE10-\uFE19"""+"""\uFE30-\uFE6F"""+"""\uFF00-\uFF60"""+"""\uFFE0-\uFFE6"""+// scalastyle:on nonascii"""]""").r println(fullWidthRegex.findAllIn(s1).size) //2
This regular expression is obtained experimentally under a specific font.
I don't understand what you are going to do.
How about some additional overheads when calling showString as compared to showString w/o this patch?
I tested a Dataset consisting of 100 rows, each row has two columns, one column is the index (0-99), and the other column is a random string of length 100 characters, and then the showString display is called separately.
The original showString method (w/o this patch) took about 42ms, and the improved time took about 46ms, and the performance was about 10% worse.

I think this is fine. Just copy a summary of your comments here into the comments in the code. Yes this has nothing to do with UTF8 encoding directly. You are matching UCS2 really, 16bit char values.

Do I need to merge the above commited into one commit,
Or add another new commit?
Or change the last commit comments ?
@srowen

srowen · 2018-09-04T10:50:30Z

Just add a new commit with a little info about what this does and how you generated it. The commits are squashed anyway.

…blem Some characters (called full-width characters) occupy two ordinary characters (called half-width characters) width when displayed in a terminal such as Xshell. Therefore, when the Dataset.show() method is called, if there are full-width characters and half-width characters at the same time, alignment problems will occur. This commit adds a method to calculate the number of half widths in a given string, so that it can get the correct fill when calling show method, which gives the display a more perfect alignment. Performance impact: Tested a Dataset consisting of 100 rows, each row has two columns, one column is the index (0-99), and the other column is a random string of length 100 characters, and then the showString display is called separately. The original show method (w/o this patch) took about 42ms, and the improved time took about 46ms, the performance was about 10% worse.

xuejianbest · 2018-09-04T12:36:58Z

I see, Thinks.

srowen · 2018-09-06T14:37:51Z

Merging to master and 2.4, though for some reason I can't back-port with the merge script right now. Will try later.

srowen · 2018-09-06T17:22:25Z

@dongjoon-hyun could I ask a favor; can you try merging this? the merge script should say it's merged and offer to back-port to 2.4. However I keep getting errors about it not being mergeable, although all upstream branches and my clone all are up to date.

dongjoon-hyun · 2018-09-06T17:26:43Z

@srowen . Sure. I'll try to backport this.

dongjoon-hyun · 2018-09-06T17:40:57Z

Ur, it seems that I'm facing the same situation. In this case, should we just cherry-pick to branch-2.4?

…alignment problem This is not a perfect solution. It is designed to minimize complexity on the basis of solving problems. It is effective for English, Chinese characters, Japanese, Korean and so on. ```scala before: +---+---------------------------+-------------+ |id |中国 |s2 | +---+---------------------------+-------------+ |1 |ab |[a] | |2 |null |[中国, abc] | |3 |ab1 |[hello world]| |4 |か行きゃ(kya) きゅ(kyu) きょ(kyo) |[“中国] | |5 |中国（你好）a |[“中（国）, 312] | |6 |中国山(东)服务区 |[“中(国）] | |7 |中国山东服务区 |[中(国)] | |8 | |[中国] | +---+---------------------------+-------------+ after: +---+-----------------------------------+----------------+ |id |中国 |s2 | +---+-----------------------------------+----------------+ |1 |ab |[a] | |2 |null |[中国, abc] | |3 |ab1 |[hello world] | |4 |か行きゃ(kya) きゅ(kyu) きょ(kyo) |[“中国] | |5 |中国（你好）a |[“中（国）, 312]| |6 |中国山(东)服务区 |[“中(国）] | |7 |中国山东服务区 |[中(国)] | |8 | |[中国] | +---+-----------------------------------+----------------+ ``` ## What changes were proposed in this pull request? When there are wide characters such as Chinese characters or Japanese characters in the data, the show method has a alignment problem. Try to fix this problem. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) ![image](https://user-images.githubusercontent.com/13044869/44250564-69f6b400-a227-11e8-88b2-6cf6960377ff.png) Please review http://spark.apache.org/contributing.html before opening a pull request. Closes#22048 from xuejianbest/master. Authored-by: xuejianbest <384329882@qq.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

srowen · 2018-09-06T17:49:05Z

OK, it isn't just me. We'll see if it's a broader problem.

I manually cherry-picked to branch-2.4

xuejianbest added 2 commits August 9, 2018 11:11

Merge branch 'master' into master
9aec12f

Modifying regular expressions for matching narrow characters
906c0ba

xuejianbest changed the title ~~Fix the show method to display the wide character alignment problem~~[SPARK-25108]Fix the show method to display the wide character alignment problemAug 14, 2018

xuejianbest changed the title ~~[SPARK-25108]Fix the show method to display the wide character alignment problem~~[SPARK-25108][SQL] Fix the show method to display the wide character alignment problemAug 14, 2018

srowen reviewed Aug 19, 2018
View reviewed changes

Fix the show method to display full width characters.
da37d2e
Modify regular expressions to make them more precise. Modify variable names and add comment.

srowen reviewed Aug 28, 2018
View reviewed changes

Write a utility method to get the number of half width of string
8737671
Putting `fullWidthRegex` outside `stringHalfWidth` is to reduce the cost of compiling regular expressions. Since compilation is expensive, frequently used Regexes should be constructed once.

maropu reviewed Aug 29, 2018
View reviewed changes

Move the stringHalfWidth method into util.Utils
697ac04
Move the stringHalfWidth method into util.Utils and add tests for it.

srowen requested changes Aug 29, 2018
View reviewed changes

maropu reviewed Aug 29, 2018
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/util/Utils.scala

maropu reviewed Aug 29, 2018
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/util/Utils.scala

xuejianbest added 3 commits August 30, 2018 10:14

Formatted code and non-functional changes
3d65e6b

Add tests in DatasetSuite
363de6b

Modified variable name in DatasetSuite
3649de5

srowen approved these changes Aug 30, 2018
View reviewed changes

Adding scalastyle:off nonascii in code
45ac272
To solve style errors because of include non-ascii characters in files. Disable the check for sections of code where it's appropriate to use unicode chars.

gatorsmile reviewed Aug 31, 2018
View reviewed changes

srowen approved these changes Sep 5, 2018
View reviewed changes

asfgit closed this in f5817d8Sep 6, 2018

Uh oh!

Conversation

xuejianbest commented Aug 9, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

kiszk commented Aug 9, 2018

Uh oh!

kiszk commented Aug 9, 2018

Uh oh!

xuejianbest commented Aug 10, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kiszk commented Aug 13, 2018

Uh oh!

xuejianbest commented Aug 14, 2018

Uh oh!

xuejianbest commented Aug 14, 2018

Uh oh!

kiszk commented Aug 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

srowen commented Aug 26, 2018

Uh oh!

xuejianbest commented Aug 28, 2018

Uh oh!

kiszk commented Aug 28, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuejianbestAug 29, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuejianbest commented Aug 29, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maropu commented Aug 29, 2018

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 30, 2018

Uh oh!

xuejianbest commented Aug 31, 2018

Uh oh!

xuejianbest commented Aug 31, 2018

Uh oh!

srowen commented Aug 31, 2018

Uh oh!

maropu commented Aug 31, 2018

Uh oh!

maropu commented Aug 31, 2018

Uh oh!

xuejianbest commented Aug 9, 2018•
edited
Loading

xuejianbest commented Aug 10, 2018•
edited
Loading

xuejianbestAug 29, 2018•
edited
Loading

xuejianbestSep 4, 2018•
edited
Loading