feat: implement native native existence join support

[coderfender]

Which issue does this PR close?

Closes#3881 .

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[comphead]

Thanks @coderfender this would also need to be tested by sql tests trying to cover as much edge cases as possible. Mark joins are pretty recently added in DF and might require extra testing

[coderfender]

Absolutely agreed! Thank you for the review @comphead .

[coderfender]

Mixed results with benches

Running benchmark: exists OR predicate (BHJ)
Running case: Spark
Stopped after 29 iterations, 2015 ms
Running case: Comet
Stopped after 61 iterations, 2011 ms
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (BHJ): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 43 70 15 24.4 41.1 1.0X
Comet 29 33 3 35.7 28.0 1.5X
Running benchmark: exists OR predicate (SHJ)
Running case: Spark
Stopped after 53 iterations, 2029 ms
Running case: Comet
Stopped after 44 iterations, 2043 ms
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SHJ): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 35 38 2 29.7 33.7 1.0X
Comet 42 46 5 24.8 40.3 0.8X
Running benchmark: exists OR predicate (SMJ)
Running case: Spark
Stopped after 54 iterations, 2034 ms
Running case: Comet
Stopped after 35 iterations, 2019 ms
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SMJ): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 35 38 2 29.8 33.6 1.0X
Comet 54 58 4 19.4 51.6 0.7X

[andygrove]

Thanks @coderfender. The posted benchmarks show that this would be a performance regression compared to Spark? Did performance improve since those were posted?

[comphead]

Apparently sooner or later we would need to use DF mark joins, they can be disabled by default and we can improve them in background

[coderfender]

@andygrove thank you for the comment, Yes the bechmarks are indeed different (with atleast 2x speedup)

Running benchmark: exists OR predicate (BHJ)
Running case: Spark
Stopped after 3 iterations, 2120 ms
Running case: Comet
Stopped after 7 iterations, 2043 ms
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (BHJ): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 703 707 7 1.5 670.1 1.0X
Comet 286 292 6 3.7 273.2 2.5X
Running benchmark: exists OR predicate (SHJ)
Running case: Spark
Stopped after 3 iterations, 2216 ms
Running case: Comet
Stopped after 7 iterations, 2205 ms
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SHJ): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 738 739 1 1.4 703.5 1.0X
Comet 309 315 7 3.4 294.4 2.4X
Running benchmark: exists OR predicate (SMJ)
Running case: Spark
Stopped after 3 iterations, 2181 ms
Running case: Comet
Stopped after 6 iterations, 2017 ms
OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Mac OS X 26.3.2
Apple M5 Pro
exists OR predicate (SMJ): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 714 727 15 1.5 680.9 1.0X
Comet 331 336 4 3.2 315.8 2.2X

[comphead]

nice PR in DF in terms of performance apache/datafusion#23870