[SPARK-39958][SQL] Add warning log when unable to load custom metric object

[viirya]

What changes were proposed in this pull request?

Currently when SQLAppStatusListener cannot load custom metric object, it silently ignores the error. We better provide a warning log for users.

Why are the changes needed?

Improving user experience.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests

[viirya]

cc @dongjoon-hyun

[pan3793]

Previous discussion: #31451 (comment)

I think we don't need to construct the instance using reflection, how about removing constructor checks?

[cloud-fan]

I think we don't need to construct the instance using reflection

Then how can we construct the instance?

[viirya]

Just about to ask same question.

[pan3793]

Oh, I see. SQLMetrics.createV2CustomMetric only store the class name instead of the instance itself, then constructing the instance by reflection is necessary.

Previous I misunderstand that instances constructed by Scan#supportedCustomMetrics will be passed and reused.

Could you please enhance the comment of CustomMetric to mention the constructor requirements as well?

[viirya]

Could you please enhance the comment of CustomMetric to mention the constructor requirements as well?

Okay.

[dongjoon-hyun]

nit. . at the end of the sentence?

[viirya]

Added. Thanks.

[dongjoon-hyun]

+1, LGTM.

[pan3793]

Utils#loadExtensions also allows "single-argument constructor that accepts SparkConf", do we need to mention that?

[viirya]

I don't think we should. Actually we don't need the implementing class to implement it.

[pan3793]

Thanks, LGTM

[viirya]

The known and unrelated sparkr failure:

Status: 1 WARNING, 2 NOTEs
See
‘/__w/spark-1/spark-1/R/SparkR.Rcheck/00check.log’
for details.
+ popd
Had CRAN check errors; see logs.
[error] running /__w/spark-1/spark-1/R/run-tests.sh ; received return code 255

Unrelated Python linter:

Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.9/dist-packages/black/__main__.py", line 3, in <module>
patched_main()
File "/usr/local/lib/python3.9/dist-packages/black/__init__.py", line 1372, in patched_main
patch_click()
File "/usr/local/lib/python3.9/dist-packages/black/__init__.py", line 1358, in patch_click
from click import _unicodefun
ImportError: cannot import name '_unicodefun' from 'click' (/usr/local/lib/python3.9/dist-packages/click/__init__.py)
Please run 'dev/reformat-python' script.

[viirya]

Merged, thanks!