Skip to content

[sqllogictest] port tests in avro.rs to sqllogictest - #6362

Merged
alamb merged 6 commits into
apache:mainfrom
e1ijah1:fix/6299/port-avro-test
May 19, 2023
Merged

[sqllogictest] port tests in avro.rs to sqllogictest#6362
alamb merged 6 commits into
apache:mainfrom
e1ijah1:fix/6299/port-avro-test

Conversation

@e1ijah1

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#6299

Rationale for this change

#6195

What changes are included in this PR?

Port the Rust unit tests from avro.rs, excluding avro_query_multiple_files, to sqllogictest.

Are these changes tested?

yes.

Are there any user-facing changes?

@github-actionsgithub-actionsBot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels May 16, 2023

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution @e1ijah1 -- this is looking very close. I left some comments about how to port the last test. Thank you 🙏

}

#[tokio::test]
async fn avro_query_multiple_files() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that this test was not ported directly -- since the avro code uses the same codepath as all the other listing tables for handling multiple files it is probably ok.

However, I think it is important coverage to maintain as the end to end coverage makes sure everything is hooked up correctly.

I think the issue is there is no way to setup these files using sqllogictest. Perhaps you can add a special sqllogictest test setup (to make the directory with multiple files) following this model:

https://github.com/apache/arrow-datafusion/blob/b578c5819fc05801f2972dd427fbc13cf4773ea8/datafusion/core/tests/sqllogictests/src/main.rs#L171

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering that avro_query_multiple_files utilizes a temporary directory, could we maintain the temporary directory within SessionContext to facilitate the creation of necessary data during the setup phase?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I think adding something to SessionContext for testing only is likely not the best idea

Maybe we could return the TempDir alongside the SessionContext like:

structTestContext{/// Context for running queriesctx:SessionContext,/// Temporary directory created and cleared at the end of the testtmpdir:TempDir}

And then change

asyncfn context_for_test_file(relative_path:&Path) -> SessionContext{

to

asyncfn context_for_test_file(relative_path:&Path) -> TestContext{

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for taking the time to review my PR 🙏. I have updated the code. Please take another look.

@e1ijah1
e1ijah1 requested a review from alambMay 18, 2023 14:36

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great -- thank you @e1ijah1

Comment threaddatafusion/core/tests/sqllogictests/src/setup.rs
Comment threaddatafusion/core/tests/sqllogictests/test_files/avro.slt
@alamb

Copy link
Copy Markdown
Contributor

This PR was failing on CI: https://github.com/apache/arrow-datafusion/actions/runs/5010297388/jobs/8991828477 -- I pushed some fixes for cargo fmt and the hash collisions failures. Hopefully the tests pass this time

@alamb
alamb merged commit d01002c into apache:mainMay 19, 2023
@alamb

Copy link
Copy Markdown
Contributor

Thanks again @e1ijah1

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratesqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port tests in avro.rs to sqllogictest

2 participants

@e1ijah1@alamb