[SPARK-32907][ML][PYTHON] Adaptively blockify instances - AFT,LiR,LoR

[zhengruifeng]

What changes were proposed in this pull request?

use maxBlockSizeInMB instead of blockSize (#rows) to control the stacking of vectors;

Why are the changes needed?

the performance gain is mainly related to the nnz of block.

Does this PR introduce any user-facing change?

yes, param blockSize -> blockSizeInMB in master

How was this patch tested?

updated testsuites

[SparkQA]

Test build #131000 has finished for PR 30355 at commit f8782a4.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35606/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35606/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35634/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35634/

[SparkQA]

Test build #131028 has finished for PR 30355 at commit 5f9e70f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[WeichenXu123]

Let's document the default 0.0 behavior (set optimal block size, current 1MB) in the HasMaxBlockSizeInMB param doc.
otherwise LGTM.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35855/

[SparkQA]

Test build #131252 has finished for PR 30355 at commit 48b2814.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35855/

[SparkQA]

Test build #131268 has finished for PR 30355 at commit 317bde9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35871/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35871/

[WeichenXu123]

LGTM.

[WeichenXu123]

Merge to master.

[zhengruifeng]

@WeichenXu123 Thank you so much!