Skip to content

[SPARK-48941][PYTHON][ML] Replace RDD read / write API invocation with Dataframe read / write API - #47411

Closed
WeichenXu123 wants to merge 4 commits into
apache:masterfrom
WeichenXu123:SPARK-48909-follow-up
Closed

[SPARK-48941][PYTHON][ML] Replace RDD read / write API invocation with Dataframe read / write API #47411
WeichenXu123 wants to merge 4 commits into
apache:masterfrom
WeichenXu123:SPARK-48909-follow-up

Conversation

@WeichenXu123

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

PysparkML: Replace RDD read / write API invocation with Dataframe read / write API

Why are the changes needed?

Follow-up of #47341

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

Was this patch authored or co-authored using generative AI tooling?

No.

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
@HyukjinKwonHyukjinKwon changed the title [SPARK-48941] PysparkML: Replace RDD read / write API invocation with Dataframe read / write API [SPARK-48941][PYTHON][ML] Replace RDD read / write API invocation with Dataframe read / write API Jul 19, 2024
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
Comment threadpython/pyspark/ml/util.py Outdated
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
@WeichenXu123

Copy link
Copy Markdown
ContributorAuthor

Merged to master.

jingz-db pushed a commit to jingz-db/spark that referenced this pull request Jul 22, 2024
…h Dataframe read / write API
### What changes were proposed in this pull request?
PysparkML: Replace RDD read / write API invocation with Dataframe read / write API
### Why are the changes needed?
Follow-up of apache#47341
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closesapache#47411 from WeichenXu123/SPARK-48909-follow-up.
Authored-by: Weichen Xu <weichen.xu@databricks.com>
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@WeichenXu123@HyukjinKwon@zhengruifeng