Uh oh!
There was an error while loading. Please reload this page.
[SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic - #46155
Closed
HyukjinKwon wants to merge 8 commits into
Closed
[SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic#46155HyukjinKwon wants to merge 8 commits into
HyukjinKwon wants to merge 8 commits into
Conversation
HyukjinKwonforce-pushed
the
SPARK-47933
branch
6 times, most recently
from
April 22, 2024 06:28
5fa46be to
a6677c5Comparezhengruifeng
approved these changes
Apr 22, 2024
HyukjinKwon
commented
Apr 22, 2024
MemberAuthor
Let me run this naming stuff separately (maybe sending an email to the dev mailing list) because Spark Classic has been used in other places lately in any event. |
dongjoon-hyun
commented
Apr 22, 2024
Member
Sure, please proceed it. We can discuss the naming separately, @HyukjinKwon . |
ueshin
reviewed
Apr 22, 2024
| # | ||
| from pyspark.ml import functions as PyMLFunctions | ||
| from pyspark.sql.connect.column import Column | ||
| from pyspark.sql.column import Column |
Member
There was a problem hiding this comment.
Should keep it as-is to be consistent with the classic functions.py if it changes to refer classic?
MemberAuthor
There was a problem hiding this comment.
It is actually related to the type hints. We're using pyspark.sql.Column for type hints, and when we actually need to create the instance, we should use pyspark.sql.classic.column import Column or pyspark.sql.connect.column import Column .. it is also a bit messy. I should probably clean those up separately.
| from pyspark.ml.linalg import Matrix, Vector | ||
| from pyspark.ml.wrapper import JavaWrapper, _jvm | ||
| from pyspark.sql.column import Column, _to_seq | ||
| from pyspark.sql.column import Column |
Member
There was a problem hiding this comment.
Is this consistent with functions.py?
Member
There was a problem hiding this comment.
or import _to_seq later as the other files do?
MemberAuthor
There was a problem hiding this comment.
This file is only used in Spark Classic so it should probably be fine. It is a bit messy and complicated .. I will try to refactor them separately.
HyukjinKwon
commented
Apr 23, 2024
MemberAuthor
Merged to master. For naming, I will send an email soon. |
HyukjinKwon added a commit
that referenced
this pull request
Apr 24, 2024
…t `_with_origin` ### What changes were proposed in this pull request? This PR is a followup of #46155 that adds check of `__name__` at `_with_origin`. ### Why are the changes needed? It is possible for a callable instance without __name__ attribute or/and __module__ attribute to be wrapped. For example, functools.partial. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `./bin/pyspark` ### Was this patch authored or co-authored using generative AI tooling? No. Closes#46198 from HyukjinKwon/SPARK-47933-followup2. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
dongjoon-hyun pushed a commit
that referenced
this pull request
Apr 25, 2024
…n `pyspark-connect` ### What changes were proposed in this pull request? This PR is a followup of #46155 that removes the reference of `_to_seq` that `pyspark-connect` package does not have. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8821919392/job/24218893631 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released out yet. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#46229 from HyukjinKwon/SPARK-47933-followuptmp. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
HyukjinKwon added a commit
that referenced
this pull request
Apr 28, 2024
…` reference in `pyspark.ml.stat` ### What changes were proposed in this pull request? This PR is a followup of #46155 that removes the reference of `_to_seq` that `pyspark-connect` package does not have. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8861971303 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released out yet. ### How was this patch tested? Manually tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#46262 from HyukjinKwon/SPARK-47933-followup4. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon added a commit
that referenced
this pull request
Apr 30, 2024
… classes ### What changes were proposed in this pull request? This PR proposes to fix `pyspark-connect` build that is broken by #46155. It is similar with the followups such as #46229 and #46262 but make sure the tests pass within this PR. ### Why are the changes needed? To recover the CI https://github.com/apache/spark/actions/runs/8884003967 ### Does this PR introduce _any_ user-facing change? No, the main change has not been released yet. ### How was this patch tested? CI in my own fork: https://github.com/HyukjinKwon/spark/actions/runs/8889698252/job/24408487227 ### Was this patch authored or co-authored using generative AI tooling? No. Closes#46294 from HyukjinKwon/SPARK-48052. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Same as #46129 but for
Columnclass.Why are the changes needed?
Same as #46129
Does this PR introduce any user-facing change?
Same as #46129
How was this patch tested?
Manually tested, and CI should verify them.
Was this patch authored or co-authored using generative AI tooling?
No.