Skip to content

Update to object_store 0.6 and arrow 41 - #6374

Merged
tustvold merged 8 commits into
apache:mainfrom
tustvold:object-store-0.6
Jun 6, 2023
Merged

Update to object_store 0.6 and arrow 41#6374
tustvold merged 8 commits into
apache:mainfrom
tustvold:object-store-0.6

Conversation

@tustvold

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

The next release of object_store contains a number of breaking changes, lets verify them before cutting the release

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added the core Core DataFusion crate label May 17, 2023
builder = builder.with_token(session_token);
}
} else {
let config = aws_config::from_env().load().await;

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike before, this now ensures that users get the full coherent configuration experience that they might expect from a CLI tool.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @r4ntix as I think you contributed some of the previous configuration code

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @r4ntix as I think you contributed some of the previous configuration code

@alamb Thanks for the heads up, I think this is a great improvement. Thanks @tustvold 👏

@tustvold

Copy link
Copy Markdown
ContributorAuthor

Waiting for apache/arrow-rs#4251

@tustvoldtustvold changed the title Prepare for object_store 0.6Prepare for object_store 0.6 and arrow 41Jun 2, 2023
@github-actionsgithub-actionsBot added the substrait Changes to the substrait crate label Jun 2, 2023
@tustvoldtustvold changed the title Prepare for object_store 0.6 and arrow 41Update to object_store 0.6 and arrow 41Jun 6, 2023
@tustvold
tustvold marked this pull request as ready for review June 6, 2023 07:36

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @tustvold

ctx.register_listing_table(
"my_table",
&format!("file://{testdata}"),
&format!("file://{testdata}/alltypes_plain.parquet"),

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nice validation of https://github.com/apache/arrow-rs/issues/4230 being a valuable change. This test was broken, as the lack of a trailing / meant it wouldn't list the directory, and this meant it silently failed to list any files. Schema inference fails on the root directory (correctly) so I changed this to a single file

@tustvold
tustvold merged commit 91e75d7 into apache:mainJun 6, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratesubstraitChanges to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@tustvold@r4ntix@alamb