Skip to content

Update arrow 49.0.0 and object_store 0.8.0 - #8029

Merged
tustvold merged 11 commits into
apache:mainfrom
tustvold:prepare-object-store-0.8.0
Nov 18, 2023
Merged

Update arrow 49.0.0 and object_store 0.8.0#8029
tustvold merged 11 commits into
apache:mainfrom
tustvold:prepare-object-store-0.8.0

Conversation

@tustvold

@tustvoldtustvold commented Nov 2, 2023

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#8092

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate labels Nov 2, 2023
Comment threaddatafusion/core/tests/fifo.rs Outdated

/// It tests the INSERT INTO functionality.
#[tokio::test]
#[ignore]

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#8021 shows how these tests might be ported

@tustvoldtustvold changed the title Prepare object store 0.8.0Update arrow 49.0.0 and object_store 0.8.0Nov 7, 2023
@tustvold
tustvoldforce-pushed the prepare-object-store-0.8.0 branch from e60aa4d to d0875ceCompareNovember 7, 2023 17:24
@alamb

alamb commented Nov 8, 2023

Copy link
Copy Markdown
Contributor

I believe this upgrade is blocked on sorting out how to upgrade the design to handle FIFO files as described in #8021

@tustvold
tustvold marked this pull request as ready for review November 17, 2023 22:45
@alamb

Copy link
Copy Markdown
Contributor

Closes #8092

Technically speaking this bug was fixed by the update to 48.0.1

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @tustvold -- I know this was a long time in the making but the result is great.

I think this upgrade makes the code look much nicer (removing the double layer of errors from the object store APIs is quite nice 👨‍🍳 👌 )

.await?
.try_collect()
.await?;
let entries: Vec<_> = self.store.list(Some(&self.path)).try_collect().await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that is certainly nicer

let schema = Schema::new(vec![Field::new("a", DataType::Int32, false)]);

let result = cast(col("a", &schema).unwrap(), &schema, DataType::LargeBinary);
let result = cast(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change needed because arrow now supports casting Int32 --> LargeBinary?

@tustvold
tustvold merged commit a984f08 into apache:mainNov 18, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratephysical-exprChanges to the physical-expr cratessqllogictestSQL Logic Tests (.slt)substraitChanges to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet error: Invalid offset in sparse column chunk data

2 participants

@tustvold@alamb