The DictionaryFilter patch included a change to reuse a ParquetFileReader rather than creating new ones (and opening new file streams). But by reusing a reader, the projection columns are no longer set. The result is that all columns are read from disk, but only the requested columns are materialized. The fix is to set the requested schema on the file reader when it is fetched from the read context inside InternalParquetRecordReader.
Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue
Related issues:
PRs and other links:
Note: This issue was originally created as PARQUET-751. Please see the migration documentation for further details.
The
DictionaryFilterpatch included a change to reuse aParquetFileReaderrather than creating new ones (and opening new file streams). But by reusing a reader, the projection columns are no longer set. The result is that all columns are read from disk, but only the requested columns are materialized. The fix is to set the requested schema on the file reader when it is fetched from the read context insideInternalParquetRecordReader.Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue
Related issues:
PRs and other links:
Note: This issue was originally created as PARQUET-751. Please see the migration documentation for further details.