Describe the enhancement requested
The asof-join has a big inefficiency when emitting columns from the left hand side of the join. Let me explain it visually, using a simple example from pandas.merge_asof, the equivalent to Arrow's asof-join. Pandas is of course a completely separate library, but this example is small and illustrative.
>>> left = pd.DataFrame({"a": [1, 5, 10], "left_val": ["a", "b", "c"]})
>>> left
a left_val
0 1 a
1 5 b
2 10 c
>>> right = pd.DataFrame({"a": [1, 2, 3, 6, 7], "right_val": [1, 2, 3, 6, 7]})
>>> right
a right_val
0 1 1
1 2 2
2 3 3
3 6 6
4 7 7
>>> pd.merge_asof(left, right, on="a")
a left_val right_val
0 1 a 1
1 5 b 3
2 10 c 7
Notice how
- the output has the same # of rows as the left hand table.
- more than that, the columns stemming from the LHS are simply contiguous slices of the LHS input table
In Arrow's implementation, we emit output arrays by copying the data (code pointer) cell-by-cell. This is necessary for the right hand side, where you may reuse or drop rows entirely.
However, for the left hand side, it'd be much faster and lower memory to just make zero copy Array::Slices
Component(s)
C++
Describe the enhancement requested
The asof-join has a big inefficiency when emitting columns from the left hand side of the join. Let me explain it visually, using a simple example from pandas.merge_asof, the equivalent to Arrow's asof-join. Pandas is of course a completely separate library, but this example is small and illustrative.
Notice how
In Arrow's implementation, we emit output arrays by copying the data (code pointer) cell-by-cell. This is necessary for the right hand side, where you may reuse or drop rows entirely.
However, for the left hand side, it'd be much faster and lower memory to just make zero copy Array::Slices
Component(s)
C++