Skip to content

CI: Windows test execution time - #13710

Closed
comphead wants to merge 54 commits into
apache:mainfrom
comphead:dev
Closed

CI: Windows test execution time#13710
comphead wants to merge 54 commits into
apache:mainfrom
comphead:dev

Conversation

@comphead

@compheadcomphead commented Dec 10, 2024

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Related to #13726
Closes #.

Rationale for this change

Test Windows compilation time improvements

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added the development-process Related to development process of DataFusion label Dec 10, 2024
@comphead

Copy link
Copy Markdown
ContributorAuthor
Executed "map.slt". Took 460.5636141s
Executed "window.slt". Took 217.240136s
Executed "struct.slt". Took 1316.049712s
Executed "array.slt". Took 2407.7054095s

Thats really weird, a simple array SLT takes 40 mins

@compheadcomphead changed the title CI: Windows compilation timeCI: Windows test execution timeDec 10, 2024
@korowa

Copy link
Copy Markdown
Contributor

@comphead disregarding build arguments, which may affect build performance, there seems to be a known issue with IO on windows runners which is noticeably slower than linux/mac ones. From comments to that issue it looks like some projects are able to get performance benefit using DevDrive, so it may be a different perspective to look at this problem from.

@comphead

Copy link
Copy Markdown
ContributorAuthor

@comphead disregarding build arguments, which may affect build performance, there seems to be a known issue with IO on windows runners which is noticeably slower than linux/mac ones. From comments to that issue it looks like some projects are able to get performance benefit using DevDrive, so it may be a different perspective to look at this problem from.

Thanks @korowa for the hint I was playing with lots of combination including different Windows images but nothing seems to work.

@comphead

Copy link
Copy Markdown
ContributorAuthor

Some projects give up rust testing on Windows like DataDog/orchestrion#415

@alamb wondering should we pause the Windows flow temporarily? The build time is incredibly slow and we holding GH resources for way longer time

@alamb

Copy link
Copy Markdown
Contributor

Another thing that would benefit the project as a whole is to reduce build time.

One thing I noticed a while ago is that each binary built (e.g. every example) was around 100MB

I was thinking that we could reduce / conslidate the number of examples / binaries built which would speed up all runners (not just Windows)

For example consolidating https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/regexp.rs and other examples of using functions into dataframe_api.rs might be a good win

@comphead

Copy link
Copy Markdown
ContributorAuthor

Closing it as there is no any progress after all the experiments

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-processRelated to development process of DataFusion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@comphead@korowa@alamb