Skip to content

Read out only the required columns from a Feather file on Disk #28827

Description

@asfimport

The LZ4 compressed feather format works really well and is quite nice and fast than Parquet when reading all the columns out. For single columns, looks like feather format does not yet support the capability to read out only the required columns from the disk. Are there any plans to add support for this? Here are some numbers to support my claim from an experiment with a table with 17 columns and 5 million rows in uncompressed parquet and LZ4 compressed feather format. No memory mapping involved.


pq_all_cols: 0.4179724836349487 ms
feather_all_cols: 0.26202451705932617 ms
pq_single_col: 0.10951032638549804 ms
feather_single_col: 0.2119576358795166 ms

Reporter: Jayjeet Chakraborty / @JayjeetAtGithub

Related issues:

Note: This issue was originally created as ARROW-13126. Please see the migration documentation for further details.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions