[C++] ScanNode takes filter but doesn't filter

[asfimport]

This turned out to be my confusion in ARROW-13344: because the ScanNode takes a filter and projection, I wasn't creating a FilterNode because I assume that the filter is already applied--why else would I provide a filter to the ScanNode? But it turns out that if you don't Filter again, you get unfiltered results:

Table$create(
 group=c(1, 2), 
 value=c(5, 6)
) %>% 
 filter(value > 5) %>% 
 group_by(group) %>% 
 summarize(sum(value)) %>% 
 collect()

# A tibble: 2 x 2
 group `sum(value)`
 <dbl> <dbl>
1 1 5
2 2 6

Reporter: Neal Richardson / @nealrichardson

Related issues:

• [C++] ScanNode is not using the filter expression (is duplicated by)
• [R] Initial bindings for ExecPlan/ExecNode (relates to)

_{Note: This issue was originally created as ARROW-13498. Please see the migration documentation for further details.}

[asfimport]

Ben Kietzman / @bkietz:
This is as-designed: the ScanNode only uses the filter for predicate pushdown- so in the context of datasets, you would exclude some files whose partition expression contradicted the filter. It does not remove all rows excluded by the filter- that is deferred to FilterNode.

Similarly, ScanNode does not perform projection- it ensures that fields referenced by the projection are loaded from disk (which for example in the case of parquet means you will pay less IO if you reference fewer fields).

[asfimport]

Weston Pace / @westonpace:
Does that mean it might make sense to have three filter spots? The pushdown filter, a filter before projection, and a filter after projection?

[asfimport]

Ben Kietzman / @bkietz:
I'm not sure when we'd have filter after projection. We might have a filter after a grouped aggregation (analagous to a SQL HAVING clause)

[asfimport]

Neal Richardson / @nealrichardson:
Sounds to me like implementation details are leaking into the interface. Or, maybe ExecNodes aren't the interface we want--sounds like they're not quite a logical plan but also not quite a physical plan.

Also, I can't project and then filter on the thing I projected?

[asfimport]

Ben Kietzman / @bkietz:
ExecNodes certainly correspond to a physical plan rather than a logical plan and we'll definitely want something more declarative for user facing APIs- eventually.

You can certainly project then filter on what you've projected, I'm just not sure when that'd be preferable to a single filter pass.