Search before asking
Description
DUP_KEYS MemTables currently create one shared_ptr<RowInBlock> for every inserted row, even though these rows only require their positions for sorting.
This introduces substantial per-row metadata overhead, frequent heap allocations, and unnecessary allocation churn during large data loads. The overhead becomes
especially significant when a MemTable contains a large number of rows.
We can use a more compact representation for DUP_KEYS while preserving the existing sorting semantics:
- Sort rows by key in ascending order.
- For equal keys, sort by row position in descending order.
- Keep the existing behavior for UNIQUE_KEYS and AGG_KEYS unchanged.
Solution
Replace the per-row shared_ptr<RowInBlock> objects used by DUP_KEYS MemTables with a contiguous uint32_t row-position vector.
The row positions are reserved before rows are appended to the mutable block and populated using std::iota, avoiding additional allocations after the block has been
modified. The existing RowInBlock representation remains unchanged for UNIQUE_KEYS and AGG_KEYS because those models require aggregation state.
This optimization reduces explicit row-index metadata from at least dozens of bytes per row to 4 bytes per row, eliminates per-row object allocations, and improves
MemTable insertion efficiency without changing query or load semantics.
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
DUP_KEYS MemTables currently create one
shared_ptr<RowInBlock>for every inserted row, even though these rows only require their positions for sorting.This introduces substantial per-row metadata overhead, frequent heap allocations, and unnecessary allocation churn during large data loads. The overhead becomes
especially significant when a MemTable contains a large number of rows.
We can use a more compact representation for DUP_KEYS while preserving the existing sorting semantics:
Solution
Replace the per-row
shared_ptr<RowInBlock>objects used by DUP_KEYS MemTables with a contiguousuint32_trow-position vector.The row positions are reserved before rows are appended to the mutable block and populated using
std::iota, avoiding additional allocations after the block has beenmodified. The existing
RowInBlockrepresentation remains unchanged for UNIQUE_KEYS and AGG_KEYS because those models require aggregation state.This optimization reduces explicit row-index metadata from at least dozens of bytes per row to 4 bytes per row, eliminates per-row object allocations, and improves
MemTable insertion efficiency without changing query or load semantics.
Are you willing to submit PR?
Code of Conduct