Describe the enhancement requested
Now that we are starting to introduce formal ordering we can create an AsofJoinNode variant that works even if use_threads is true. A rough overview of the algorithm:
- Require all inputs to be sequenced on the
on field in ascending order - As each batch arrives put it into a sequenced queue. In the
process step we should accumulate the batch in an accumulation vector for each input. Then, still in the process step, we record the latest on time and store it with the batch. Then determine if we have enough data in all inputs to process. If so, we create a task to asof join all those batches. This may require a binary search into each input to make sure we cut at a clean point but that should be pretty quick.
For example
// Accumulate queues just after insert into R0
LEFT | R0 | R1
B t=300 | |
B t=150 | | B t=500
B t=100 | B t=200 | B t=20
// Task to process
LEFT | R0 | R1
B t=200 | |
B t=150 | | B t=200
B t=100 | B t=200 | B t=20
// Remaining accumulation queue state
LEFT | R0 | R1
B t=300 (sliced) | | B t=500 (sliced)
Component(s)
C++
Describe the enhancement requested
Now that we are starting to introduce formal ordering we can create an AsofJoinNode variant that works even if use_threads is true. A rough overview of the algorithm:
onfield in ascending orderprocessstep we should accumulate the batch in an accumulation vector for each input. Then, still in the process step, we record the latest on time and store it with the batch. Then determine if we have enough data in all inputs to process. If so, we create a task to asof join all those batches. This may require a binary search into each input to make sure we cut at a clean point but that should be pretty quick.For example
Component(s)
C++