[SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R by yoda-mon · Pull Request #33992 · apache/spark

yoda-mon · 2021-09-14T07:07:09Z

What changes were proposed in this pull request?

octet_length: caliculate the byte length of strings
bit_length: caliculate the bit length of strings
Those two string related functions are only implemented on SparkSQL, not on Scala, Python and R.

Why are the changes needed?

Those functions would be useful for multi-bytes character users, who mainly working with Scala, Python or R.

Does this PR introduce any user-facing change?

Yes. Users can call octet_length/bit_length APIs on Scala(Dataframe), Python, and R.

How was this patch tested?

unit tests

into add-bit-octet-length

sarutak · 2021-09-14T07:41:43Z

ok to test.

HyukjinKwon · 2021-09-14T08:48:16Z

the change looks fine otherwise.

SparkQA · 2021-09-14T08:49:20Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47752/

SparkQA · 2021-09-14T11:43:14Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47756/

yoda-mon · 2021-09-14T11:59:23Z

@HyukjinKwon @sarutak
Thank you for your comments, I fixed the commented points.

SparkQA · 2021-09-14T12:27:29Z

Test build #143259 has finished for PR 33992 at commit 545fbe2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-09-14T12:35:45Z

retest this please.

SparkQA · 2021-09-14T12:38:24Z

Test build #143249 has finished for PR 33992 at commit c0ffdce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-14T13:02:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47762/

SparkQA · 2021-09-14T13:12:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47762/

SparkQA · 2021-09-14T13:40:23Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47763/

SparkQA · 2021-09-14T13:55:24Z

Test build #143253 has finished for PR 33992 at commit 3a07337.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

LGTM otherwise. Let me leave it to @sarutak

SparkQA · 2021-09-14T17:35:23Z

Test build #143260 has finished for PR 33992 at commit 545fbe2.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-15T01:58:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47791/

SparkQA · 2021-09-15T02:08:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47791/

SparkQA · 2021-09-15T02:35:46Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47793/

SparkQA · 2021-09-15T02:44:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47793/

SparkQA · 2021-09-15T05:39:06Z

Test build #143288 has finished for PR 33992 at commit afa1700.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-15T06:22:32Z

Test build #143290 has finished for PR 33992 at commit 41656f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-09-15T07:27:37Z

Merged to master. Thanks @yoda-mon and @HyukjinKwon !

AmplabJenkins · 2021-09-15T13:14:36Z

Can one of the admins verify this patch?

…or Examples in docstring ### What changes were proposed in this pull request? This PR is a minor followup of #33992 to fix the warnings during PySpark documentation build: ``` /.../spark/python/pyspark/sql/functions.py:docstring of pyspark.sql.functions.bit_length:19: WARNING: Unexpected section title or transition. ------- /.../spark/python/pyspark/sql/functions.py:docstring of pyspark.sql.functions.octet_length:19: WARNING: Unexpected section title or transition. ------- ``` We should always have the same length of hyphens with the title. ### Why are the changes needed? To remove warnings during the documentation build and show the HTML pages correctly. ### Does this PR introduce _any_ user-facing change? This is not released yet, and only in master branch. So, no to end users. ### How was this patch tested? Manually built the docs via `make clean html` at `python/docs` directory. Closes#34196 from HyukjinKwon/SPARK-36751. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

yoda-mon added 15 commits September 10, 2021 18:23

Implement scala/python octet_length
1df379b

Implement R octet_length
6df4668

Implement bit-length functions
eb2456b

Rearange the order of octet_lengh
3e149fa

Disable scalastyle to use non-ascii charactor on tests
ff2f031

Insert newline to python docstrings to keep pylint rules
5b9c98b

add space after comma to pass lint-R
848f224

Implement scala/python octet_length
4edc918

Implement R octet_length
5370a51

Implement bit-length functions
72c9398

Rearange the order of octet_lengh
761f8b6

Disable scalastyle to use non-ascii charactor on tests
f7ace95

Insert newline to python docstrings to keep pylint rules
2ed29f5

add space after comma to pass lint-R
0858001

Merge branch 'add-bit-octet-length' ofhttps://github.com/yoda-mon/spark
c0ffdce
into add-bit-octet-length

github-actionsBot added CORE PYTHON R SQL labels Sep 14, 2021

sarutak reviewed Sep 14, 2021
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala Outdated

HyukjinKwon reviewed Sep 14, 2021
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala Outdated

HyukjinKwon reviewed Sep 14, 2021
View reviewed changes

Comment threadpython/pyspark/sql/tests/test_functions.py

HyukjinKwon reviewed Sep 14, 2021
View reviewed changes

Comment threadpython/pyspark/sql/tests/test_functions.py Outdated

HyukjinKwon reviewed Sep 14, 2021
View reviewed changes

Comment threadpython/pyspark/sql/functions.py

yoda-mon added 2 commits September 14, 2021 18:42

Delete unnecessary line in the scala test.
327d7d5

Add a short description of the test
387a92f

sarutak mentioned this pull request Sep 14, 2021
[SPARK-36683][SQL] Add new built-in SQL functions: SEC and CSC #33988
Closed

sarutak reviewed Sep 14, 2021
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala Outdated

differentiate octet_length and bit_length test name
545fbe2

HyukjinKwon approved these changes Sep 14, 2021
View reviewed changes

HyukjinKwon reviewed Sep 14, 2021
View reviewed changes

Comment threadpython/pyspark/sql/functions.pyi

yoda-mon added 3 commits September 15, 2021 10:06

Formatting decstrings
afa1700

differentiate test comments
127da47

Add bit/octet length to the api document
41656f4

sarutak approved these changes Sep 15, 2021
View reviewed changes

sarutak closed this in 0666f5cSep 15, 2021

HyukjinKwon mentioned this pull request Oct 6, 2021
[SPARK-36751][PYTHON][DOCS][FOLLOW-UP] Fix unexpected section title for Examples in docstring #34196
Closed

Uh oh!

Conversation

yoda-mon commented Sep 14, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

sarutak commented Sep 14, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

yoda-mon commented Sep 14, 2021

Uh oh!

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

sarutak commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

sarutak commented Sep 15, 2021

Uh oh!

AmplabJenkins commented Sep 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants