Uh oh!
There was an error while loading. Please reload this page.
[SPARK-48220][PYTHON] Allow passing PyArrow Table to createDataFrame() - #46529
Closed
ianmcook wants to merge 30 commits into
Closed
[SPARK-48220][PYTHON] Allow passing PyArrow Table to createDataFrame()#46529ianmcook wants to merge 30 commits into
ianmcook wants to merge 30 commits into
Conversation
ianmcook
commented
May 12, 2024
Uh oh!
There was an error while loading. Please reload this page.
ianmcook
commented
May 12, 2024
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
ianmcookforce-pushed
the
SPARK-48220
branch
8 times, most recently
from
May 13, 2024 22:14
c408d0b to
795d01aComparealippai
commented
May 14, 2024
This makes the usage so much easier, thanks! What will happen with the nanosecond timestamps? Truncated to milliseconds? |
ianmcookforce-pushed
the
SPARK-48220
branch
4 times, most recently
from
May 15, 2024 16:15
005c4f7 to
7e34472CompareMemberAuthor
Truncated to microseconds for now. |
alippai
commented
May 15, 2024
I wish we had ns support in spark 4.0 as Java, parquet, arrow etc uses it natively now, but that’s certainly a different discussion and MRs :) Thanks for the new API! |
ianmcook
marked this pull request as ready for review
May 16, 2024 07:52
ianmcookforce-pushed
the
SPARK-48220
branch
3 times, most recently
from
May 16, 2024 16:10
5275bb2 to
4cc3ca2Compareianmcook
commented
May 30, 2024
MemberAuthor
Just rebased to master |
HyukjinKwon
commented
May 30, 2024
Member
will merge once ci passes |
ianmcook
commented
May 31, 2024
MemberAuthor
Thanks @HyukjinKwon. After you merge I will rebase the small follow-up PR #46688 and prep it for review. |
HyukjinKwon
commented
Jun 2, 2024
Member
Merged to master. |
HyukjinKwon pushed a commit
that referenced
this pull request
Jun 3, 2024
### What changes were proposed in this pull request? This is a small follow-up to #46529. This adds support for some more Arrow data types: - fixed-size binary - fixed-size list - large list ### Why are the changes needed? Users who are creating Spark DataFrames from PyArrow Tables will expect it to work if their Tables contain these types of columns ### Does this PR introduce _any_ user-facing change? It will prevent an error in the case where the user has one of these types of columns in their PyArrow Table. There are no other user-facing changes. ### How was this patch tested? Tests are included. ### Was this patch authored or co-authored using generative AI tooling? No Closes#46688 from ianmcook/SPARK-48374. Authored-by: Ian Cook <ianmcook@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon pushed a commit
that referenced
this pull request
Jun 16, 2024
### What changes were proposed in this pull request? This is a small follow-up to #46529. It fixes a known issue affecting PyArrow Tables passed to `spark.createDataFrame()`. After this PR, if the user is running PyArrow 17.0.0 or higher, null values in MapArray columns containing nested fields or timestamps will be preserved. ### Why are the changes needed? Before this PR, null values in MapArray columns containing nested fields or timestamps are replaced by empty lists when a PyArrow Table is passed to `spark.createDataFrame()`. ### Does this PR introduce _any_ user-facing change? It prevents loss of nulls in the case described above. There are no other user-facing changes. ### How was this patch tested? A test is included. ### Was this patch authored or co-authored using generative AI tooling? No Closes#46837 from ianmcook/SPARK-48302. Authored-by: Ian Cook <ianmcook@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
createDataFrame().toArrow().Why are the changes needed?
This seems like a logical next step after the addition of a
toArrow()DataFrame method in #45481.Does this PR introduce any user-facing change?
Users will have the ability to pass PyArrow Tables to
createDataFrame(). There are no changes to the parameters ofcreateDataFrame(). The only difference is thatdatacan now be a PyArrow Table.How was this patch tested?
Many tests were added, for Spark Classic and Spark Connect. I ran the tests locally with older versions of PyArrow installed (going back to 10.0).
Was this patch authored or co-authored using generative AI tooling?
No