[SPARK-21261][DOCS]SQL Regex document fix

[visaxin]

SQL regex docs change:
SELECT _FUNC_('100-200', '(\d+)', 'num') => SELECT _FUNC_('100-200', '(\\d+)', 'num')

[gf53520]

test this please

[srowen]

There is another example that needs the same change near the end of the file too.

[viirya]

Do we need to fix this? I remember in the doc, we use unescaped characters.

[srowen]

@viirya I'm not an expert here, but reading the docs on line 160, I think this needs to be escaped in order to be consistent with Spark 2 default behavior? my assumption was that this was just never updated.

[viirya]

Hmm, when I wrote the docs on line 160, I was suggested to use unescaped characters.

Since Spark 2.0, string literals (including regex patterns) are unescaped in our SQL parser. For example, to match "\abc", a regular expression for regexp can be "^\abc$".

Actually, you need to write like this in spark-shell:

scala> sql("SELECT like('\\\\abc', '\\\\\\\\abc')").show
+---------------+
|\abc LIKE \\abc|
+---------------+
| true|
+---------------+
scala> sql("SELECT regexp_replace('100-200', '(\\\\d+)', 'num')").show
+-----------------------------------+
|regexp_replace(100-200, (\d+), num)|
+-----------------------------------+
| num-num|
+-----------------------------------+

The behavior of Spark 2 when parsing SQL string literal reads \\\\abc as \abc and (\\\\d+) as (\d+) in spark-shell.

But in spark-sql, you write the queries like this:

spark-sql> SELECT like('\\abc', '\\\\abc');
true
Time taken: 0.061 seconds, Fetched 1 row(s)
spark-sql> SELECT regexp_replace('100-200', '(\\d+)', 'num');
num-num
Time taken: 0.117 seconds, Fetched 1 row(s)

So depending how the shell environment processes string escaping, the query looks different. In the docs, it seems to me that writing in unescaped style can avoid this confusion?

[srowen]

Is the better fix to make it clear that this example uses unescaped style @viirya ?

[viirya]

Yeah, if we can.

[visaxin]

I add spark-sql and scala to make it clear.

[gatorsmile]

@visaxin Could you address the comment?

[visaxin]

@gatorsmile Done

[gatorsmile]

scala> spark.sql("SELECT regexp_extract('100-200', '(\\d+)-(\\d+)', 1)").collect()

[gatorsmile]

Array([100])

[gatorsmile]

scala> spark.sql("SELECT regexp_replace('100-200', '(\\d+)', 'num')").collect()
Array([num-num])

[gatorsmile]

LGTM except the above three comments.

[srowen]

@visaxin this one is old but could you update it per the last review comments?

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Ping @visaxin

[HyukjinKwon]

Ping @visaxin

[srowen]

I took this over at #21808
I don't think this change is even right as it introduces scala-shell examples.