Uh oh!
There was an error while loading. Please reload this page.
Benchmarks re-org - #5676
Merged
Merged
Conversation
AdamGSforce-pushed
the
adamg/benchmarks-reorg
branch
7 times, most recently
from
December 12, 2025 17:10
fe6b48b to
8815e93CompareContributor
Benchmarks: FineWeb NVMeSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-DS SF=1 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: Clickbench on NVMESummary
Detailed Results Table
|
AdamGSforce-pushed
the
adamg/benchmarks-reorg
branch
from
December 12, 2025 18:59
a55e8a5 to
76335c9CompareAdamGS
commented
Dec 12, 2025
ContributorAuthor
End of the week state:
|
Contributor
Benchmarks: TPC-H SF=1 on NVMESummary
Detailed Results Table
|
gatesn
reviewed
Dec 15, 2025
| @@ -0,0 +1,44 @@ | |||
| [package] | |||
| name = "df-bench" | |||
Contributor
Benchmarks: Statistical and Population GeneticsSummary
Detailed Results Table
|
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
Benchmarks: FineWeb S3Summary
Detailed Results Table
|
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Contributor
Benchmarks: Random AccessSummary
|
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Contributor
Benchmarks: CompressionSummary
Detailed Results Table
|
connortsui20
commented
Dec 17, 2025
Member
Any ideas on why the benchmark measurements are very different? Is it because this has a new version of DF? |
AdamGS
commented
Dec 18, 2025
ContributorAuthor
This PR doesn't have a new DF version. |
Merged
CodSpeed Performance ReportMerging #5676 will not alter performanceComparing Summary
Footnotes
|
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Uh oh!
There was an error while loading. Please reload this page.
| "subcommand": "tpch", | ||
| "name": "TPC-H SF=10 on S3", | ||
| "local_dir": "bench-vortex/data/tpch/10.0", | ||
| "local_dir": "vortex-bench/data/tpch/10.0", |
Contributor
There was a problem hiding this comment.
shall we have this in benchmarks?
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
joseph-isaacs
commented
Dec 18, 2025
Contributor
I think we need to ensure that the way data is now generated doesn't really change the benchmarks |
AdamGS
commented
Dec 18, 2025
ContributorAuthor
@joseph-isaacs can you expand on what you mean? Right now for some benchmarks like TPC-DS we don't even have a basic correctness check. |
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
gatesn
approved these changes
Dec 19, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes two big changes to our benchmarking setup:
vx-benchwhich allows comparing various runs, persisting results and takes care of building and running the individual binaries.vx-benchis written in Python usingtyper.I've tried to not touch the CI setup too much, but it can be simplified and improved significantly to fit this structure.
Notes for reviewers
MemTablewhich is just aVec<RecordBatch>, which is now longer.I'm looking into the differences in Vortex results, not sure what's going on there.Turns out thevortex-compactresults are slower because they actually run on data compressed with the compact compressor! Some of the tables are about half the size, but they are slower to decompress/process.Seems like we've been dropping some compression benchmark results, so they dont have a baseline (or I changed the name) I'm trying to figure out what's going on there.I've changed the name of some compression benchmarks, ratios have the same name though.vx-benchdemo