[SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs by GuoPhilipse · Pull Request #29056 · apache/spark

GuoPhilipse · 2020-07-09T13:57:14Z

What changes were proposed in this pull request?

update sql-ref docs, the following key words will be added in this PR.

CASE/ELSE
WHEN/THEN
MAP KEYS TERMINATED BY
NULL DEFINED AS
LINES TERMINATED BY
ESCAPED BY
COLLECTION ITEMS TERMINATED BY
PIVOT
LATERAL VIEW OUTER?
ROW FORMAT SERDE
ROW FORMAT DELIMITED
FIELDS TERMINATED BY
IGNORE NULLS
FIRST
LAST

Why are the changes needed?

let more users know the sql key words usage

Does this PR introduce any user-facing change?

How was this patch tested?

No

update masterbranch

sync

maropu · 2020-07-10T01:28:54Z

cc: @huaxingao

maropu · 2020-07-10T01:31:43Z

Thanks for the work, @GuoPhilipse ! Please list up the keywords in the PR description that this PR will add.

GuoPhilipse · 2020-07-10T01:36:27Z

Thanks for the work, @GuoPhilipse ! Please list up the keywords in the PR description that this PR will add.
sure, will add it later.

maropu · 2020-07-17T14:07:52Z

btw, could you update the PR description, too? e.g., remove FROM

GuoPhilipse · 2020-07-17T14:08:47Z

btw, could you update the PR description, too? e.g., remove FROM

sure ,wil update soon.

huaxingao · 2020-07-17T23:13:05Z

also edit the menu-sql.yaml to update the sidebar menu?

huaxingao · 2020-07-19T03:04:02Z

@GuoPhilipse Sorry I have more nitpicks on grammar. I know this is very tedious. I went through all these when I did the doc PRs. Thanks a lot for doing this!
I have no more comments :)

GuoPhilipse · 2020-07-19T13:48:18Z

Thanks @maropu @huaxingao, really appreciated by your earnest and carefull review. I have leant a lot from this PR. Thanks for your time.
BTW @maropu do you have any new comments?

maropu · 2020-07-20T01:09:53Z

+
+```sql
+PIVOT ( { aggregate_expression [ AS aggregate_expression_alias ] } [ , ... ]
+ FOR column_list IN ( expression_list ) )


cc: @maryannxue @gatorsmile

maropu · 2020-07-21T07:19:49Z

 Specifies an aggregate function name (MIN, MAX, COUNT, SUM, AVG, etc.).
+ Note that `FIRST` and `LAST` have an optional `IGNORE NULLS` clause; when the option specified,
+ it will returns the first or last value that is not null (or null if all values are null). 
+ **Syntax:** `[ FIRST | LAST ] ( [ distinct ] expression [ IGNORE NULLS ] ) [ FILTER ( WHERE boolean_expression ) ]`


We still need **Syntax:** [ FIRST | LAST ] ( [ distinct ] expression [ IGNORE NULLS ] ) [ FILTER ( WHERE boolean_expression ) ]? It looks verbose to me though.

now have have aggregate functions above and examples in the end, looks enough,let me move it

maropu · 2020-07-21T07:26:19Z

Thanks for the work, @GuoPhilipse ! Looks ok except for one comment. Could you check for the final sign-off? @srowen @dongjoon-hyun @huaxingao @dilipbiswal Note: since all the SQL syntaxes in this update has been implemented in 3.0.0, we can merge this into branch-3.0.

maropu · 2020-07-21T07:34:10Z

@GuoPhilipse Could you generate HTML docs and check if all things are okay for this udpate?
In most PRs for doc updates, the screenshots of HTML docs are often attached in the PR description, e.g., #28672

maropu · 2020-07-21T12:02:33Z

@GuoPhilipse Btw, we still have more document improvement issues;

Improve the structure of auto-generated built-in function pages in SQL references, https://issues.apache.org/jira/browse/SPARK-31513
Add group tags (ExpressionDescription) to all the built-in functions: (Related JIRA: https://issues.apache.org/jira/browse/SPARK-31429)

As you know, we currently have the two duplicated document pages for built-in functions;

If we finish adding the tags, we might be able to remove the former page. If you're interested in more contributions, feel free to take them over. Its very helpful. cc: @HyukjinKwon @huaxingao

GuoPhilipse · 2020-07-21T12:17:07Z

@GuoPhilipse Btw, we still have more document improvement issues;
Improve the structure of auto-generated built-in function pages in SQL references, https://issues.apache.org/jira/browse/SPARK-31513
Add group tags (ExpressionDescription) to all the built-in functions: (Related JIRA: https://issues.apache.org/jira/browse/SPARK-31429)
As you know, we currently have the two duplicated document pages for built-in functions;
https://spark.apache.org/docs/latest/api/sql/index.html
https://spark.apache.org/docs/latest/sql-ref-functions.html#built-in-functions
If we finish adding the tags, we might be able to remove the former page. If you're interested in more contributions, feel free to take them over. Its very helpful. cc: @HyukjinKwon @huaxingao

Thanks @maropu , will glad to take more look on this.

GuoPhilipse · 2020-07-21T13:31:48Z

@GuoPhilipse Could you generate HTML docs and check if all things are okay for this udpate?
In most PRs for doc updates, the screenshots of HTML docs are often attached in the PR description, e.g., #28672
will append later

maropu · 2020-07-23T00:51:07Z

ok to test

maropu · 2020-07-23T00:53:32Z

Note: If nobody has comments, this is just a doc improvement, so I will merge in a few days.

SparkQA · 2020-07-23T01:07:30Z

Test build #126368 has finished for PR 29056 at commit ffa0603.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change? ![image](https://user-images.githubusercontent.com/46367746/88148830-c6dc1f80-cc31-11ea-81ea-13bc9dc34550.png) ![image](https://user-images.githubusercontent.com/46367746/88148968-fb4fdb80-cc31-11ea-8649-e8297cf5813e.png) ![image](https://user-images.githubusercontent.com/46367746/88149000-073b9d80-cc32-11ea-9aa4-f914ecd72663.png) ![image](https://user-images.githubusercontent.com/46367746/88149021-0f93d880-cc32-11ea-86ed-7db8672b5aac.png) ### How was this patch tested? No Closes#29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse <guofei_ok@126.com> Co-authored-by: GuoPhilipse <46367746+GuoPhilipse@users.noreply.github.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org> (cherry picked from commit 8de4333) Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

maropu · 2020-07-28T00:42:48Z

Merged to master/branch-3.0. Thanks, @GuoPhilipse !

gatorsmile · 2020-09-25T07:45:56Z

@@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier
 [ LOCATION path ]


The bucketSpec is still missing in CREATE HIVE FORMAT table, right?
[ CLUSTERED BY ( col_name3, col_name4, ... ) [ SORTED BY ( col_name [ ASC | DESC ], ... ) ] INTO num_buckets BUCKETS ]

Any reason we did not add it? @huaxingao @GuoPhilipse

Sorry, we missed that.
@GuoPhilipse Could you please have a follow-up to add bucketSpec? Thanks!

Yes, we need it, will add it soon.

GuoPhilipseand others added 23 commits May 21, 2020 07:36

Merge pull request#1from apache/master
bb1efa2
update masterbranch

Merge pull request#2from apache/master
1459d5b
sync

Merge pull request#3from apache/master
88c40fe
sync

Merge pull request#4from apache/master
df22083
sync

Merge pull request#5from apache/master
0436611
sync

Merge pull request#6from apache/master
ed80c84
sync

Merge pull request#7from apache/master
39ca87c
sync

Merge pull request#8from apache/master
c3b3c89
sync

Merge pull request#9from apache/master
c3546eb
sync

Merge pull request#10from apache/master
77a339a
sync

Merge pull request#11from apache/master
a6b4f74
sync

Merge pull request#12from apache/master
664277e
sync

Merge pull request#13from apache/master
fd677c9
sync

Merge pull request#14from apache/master
93b1f63
sync

Merge pull request#15from apache/master
a5b5474
sync

Merge pull request#16from apache/master
f4556a4
sync

Merge pull request#17from apache/master
6071006
sync

Merge pull request#18from apache/master
596b842
sync

Merge pull request#19from apache/master
dc24541
sync

Merge pull request#20from apache/master
c6f541b
sync

update case when docs
a80fa85

update case when docs
e4c2a17

update case when docs
24b64a8

probot-autolabelerBot added the DOCS label Jul 9, 2020

GuoPhilipse changed the title ~~[SPARK-31753][SQL][DOCS]Add missing keywords~~[SPARK-31753][SQL][DOCS][WIP]Add missing keywordsJul 9, 2020

dongjoon-hyun reviewed Jul 9, 2020
View reviewed changes

Comment threaddocs/sql-ref-syntax-qry-select-case.md Outdated

huaxingao reviewed Jul 10, 2020
View reviewed changes

Comment threaddocs/sql-ref-syntax-qry-select-case.md Outdated

update docs
b281494

huaxingao reviewed Jul 17, 2020
View reviewed changes

huaxingao reviewed Jul 18, 2020
View reviewed changes

update docs\
bed618b

maropu reviewed Jul 18, 2020
View reviewed changes

Comment threaddocs/sql-ref-syntax-ddl-create-table-hiveformat.md Outdated

huaxingao reviewed Jul 19, 2020
View reviewed changes

update docs
caa7130

maropu reviewed Jul 20, 2020
View reviewed changes

update docs
2813735

maropu reviewed Jul 21, 2020
View reviewed changes

update docs
ffa0603

maropu closed this in 8de4333Jul 28, 2020

gatorsmile reviewed Sep 25, 2020
View reviewed changes

GuoPhilipse deleted the add-missing-keywords branch September 27, 2020 11:46

		@@ -36,6 +36,14 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ] table_identifier
		[ LOCATION path ]

Uh oh!

Conversation

GuoPhilipse commented Jul 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

maropu commented Jul 10, 2020

Uh oh!

maropu commented Jul 10, 2020

Uh oh!

GuoPhilipse commented Jul 10, 2020

Uh oh!

Uh oh!

maropu commented Jul 17, 2020

Uh oh!

GuoPhilipse commented Jul 17, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huaxingao commented Jul 17, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huaxingao commented Jul 19, 2020

Uh oh!

GuoPhilipse commented Jul 19, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maropuJul 20, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maropuJul 21, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuoPhilipseJul 21, 2020

Choose a reason for hiding this comment

Uh oh!

maropu commented Jul 21, 2020

Uh oh!

maropu commented Jul 21, 2020

Uh oh!

maropu commented Jul 21, 2020

Uh oh!

GuoPhilipse commented Jul 21, 2020

Uh oh!

GuoPhilipse commented Jul 21, 2020

Uh oh!

maropu commented Jul 23, 2020

Uh oh!

GuoPhilipse commented Jul 9, 2020•
edited
Loading

maropuJul 21, 2020•
edited
Loading