[SPARK-46802][PYTHON][TESTS] Clean up obsolete code in PySpark coverage script

[HyukjinKwon]

What changes were proposed in this pull request?

This PR cleans up the obsolete code in PySpark coverage script

Why are the changes needed?

We used to use coverage_daemon.py for Python workers to track the coverage of the Python worker side (e.g., the coverage within Python UDF), added in #20204. However, seems it does not work anymore. In fact, it has been multiple years that it stopped working. The approach of replacing the Python worker itself was a bit hacky workaround. We should just get rid of them first, and find a proper way.

This should also deflake the scheduled jobs, and speed up the build.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually tested via:

./run-tests-with-coverage--python-executables=python3--testname="pyspark.sql.functions.builtin"

...
Finished test(python3): pyspark.sql.tests.test_functions (87s)
Tests passed in 87 seconds
Combining collected coverage data under
...
Creating XML report file at python/coverage.xml
Wrote XML report to coverage.xml
Reporting the coverage data at /.../spark/python/test_coverage/coverage_data/coverage
Name Stmts Miss Branch BrPart Cover
-------------------------------------------------------------------------
pyspark/__init__.py 48 7 10 3 76%
pyspark/_globals.py 16 3 4 2 75%
pyspark/accumulators.py 123 38 26 5 66%
pyspark/broadcast.py 121 79 40 3 33%
pyspark/conf.py 99 33 50 5 64%
pyspark/context.py 451 216 151 26 51%
pyspark/errors/__init__.py 3 0 0 0 100%
pyspark/errors/error_classes.py 3 0 0 0 100%
pyspark/errors/exceptions/__init__.py 0 0 0 0 100%
pyspark/errors/exceptions/base.py 91 15 24 4 83%
pyspark/errors/exceptions/captured.py 168 81 57 17 48%
pyspark/errors/utils.py 34 8 6 2 70%
pyspark/files.py 34 15 12 3 57%
pyspark/find_spark_home.py 30 24 12 2 19%
pyspark/java_gateway.py 114 31 30 12 69%
pyspark/join.py 66 58 58 0 6%
pyspark/profiler.py 244 182 92 3 22%
pyspark/rdd.py 1064 741 378 9 27%
pyspark/rddsampler.py 68 50 32 0 18%
pyspark/resource/__init__.py 5 0 0 0 100%
pyspark/resource/information.py 11 4 4 0 73%
pyspark/resource/profile.py 110 82 58 1 27%
pyspark/resource/requests.py 139 90 70 0 35%
pyspark/resultiterable.py 14 6 2 1 56%
pyspark/serializers.py 349 185 90 13 43%
pyspark/shuffle.py 397 322 180 1 13%
pyspark/sql/__init__.py 14 0 0 0 100%
pyspark/sql/catalog.py 203 127 66 2 30%
pyspark/sql/column.py 268 78 64 12 67%
pyspark/sql/conf.py 40 16 10 3 58%
pyspark/sql/context.py 170 95 58 2 47%
pyspark/sql/dataframe.py 900 475 459 40 45%
pyspark/sql/functions/__init__.py 3 0 0 0 100%
pyspark/sql/functions/builtin.py 1741 542 1126 26 76%
pyspark/sql/functions/partitioning.py 41 19 18 3 59%
pyspark/sql/group.py 81 30 32 3 65%
pyspark/sql/observation.py 54 37 22 1 26%
pyspark/sql/pandas/__init__.py 1 0 0 0 100%
pyspark/sql/pandas/conversion.py 277 249 156 2 8%
pyspark/sql/pandas/functions.py 67 49 34 0 18%
pyspark/sql/pandas/group_ops.py 89 65 22 2 25%
pyspark/sql/pandas/map_ops.py 37 27 10 2 26%
pyspark/sql/pandas/serializers.py 381 323 172 0 10%
pyspark/sql/pandas/typehints.py 41 32 26 1 15%
pyspark/sql/pandas/types.py 407 383 326 1 3%
pyspark/sql/pandas/utils.py 29 11 10 5 59%
pyspark/sql/profiler.py 80 47 54 1 39%
pyspark/sql/readwriter.py 362 253 146 7 27%
pyspark/sql/session.py 469 206 228 22 56%
pyspark/sql/sql_formatter.py 41 26 16 1 28%
pyspark/sql/streaming/__init__.py 4 0 0 0 100%
pyspark/sql/streaming/listener.py 400 200 186 1 61%
pyspark/sql/streaming/query.py 102 63 40 1 39%
pyspark/sql/streaming/readwriter.py 268 207 118 2 21%
pyspark/sql/streaming/state.py 100 68 44 0 29%
pyspark/sql/tests/__init__.py 0 0 0 0 100%
pyspark/sql/tests/test_functions.py 646 2 244 7 99%
pyspark/sql/types.py 1013 355 528 74 62%
pyspark/sql/udf.py 240 132 90 20 42%
pyspark/sql/udtf.py 152 98 52 2 33%
pyspark/sql/utils.py 160 83 54 10 45%
pyspark/sql/window.py 89 23 56 5 77%
pyspark/statcounter.py 79 58 20 0 21%
pyspark/status.py 36 13 6 0 55%
pyspark/storagelevel.py 41 9 0 0 78%
pyspark/taskcontext.py 111 63 40 1 40%
pyspark/testing/__init__.py 2 0 0 0 100%
pyspark/testing/sqlutils.py 149 44 52 1 75%
pyspark/testing/utils.py 312 238 162 2 17%
pyspark/traceback_utils.py 38 4 14 6 81%
pyspark/util.py 153 120 56 2 18%
pyspark/version.py 1 0 0 0 100%
...

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

+1, LGTM.

[HyukjinKwon]

Merged to master.