ARROW-3325: [Python][Parquet] Add "read_dictionary" argument to parquet.read_table, ParquetDataset to enable direct-to-DictionaryArray reads

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix

[wesm]

I also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

Using this option with heavily compressed data results in far less memory use and much better performance. See example benchmarks

https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

[wesm]

Here's a benchmark showing a very common "worst case" for data that dictionary-encodes very well:

Full notebook https://gist.github.com/wesm/450d85e52844aee685c0680111cbb1d7

• 1000 unique strings of length 50
• Total number of rows: 10 million
• Parquet file is 1.1MB, small enough to fit on a floppy disk

Summary:

• Using pq.read_table naive causes 516MB of memory consumption. That's almost 500x the size of the Parquet file on disk
• Using pq.read_table(data, read_dictionary=['f0']) results in only 39MB memory consumption
• The direct-dictionary read takes 106 ms on average compared with 1.8 seconds on average for the dense decoded case

[jorisvandenbossche]

Impressive benchmarks!

also added support to pyarrow.table to invoke Table.from_arrays if a list or tuple of arrays is passed. This makes for more natural code IMHO.

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

[jorisvandenbossche]

This was still being used in dask 2.1.0, released a month ago: https://github.com/dask/dask/blob/ed33fbe6ec47e361d1f6f45b84acfe0a98e511ca/dask/dataframe/io/parquet.py#L860. But, it's fixed in the latest release 2.2 released a few days ago. So it might be fine to remove, but just that we are aware it was only fixed in dask very recently.

[wesm]

Noted. This was deprecated in 0.13.0 so I think it's OK to remove since we had 2 major releases with the deprecated API and warning

[jorisvandenbossche]

Does the read_dictionary setting influence the metadata ?

[wesm]

no, I can remove this. It does change the Arrow schema though

[wesm]

done

[jorisvandenbossche]

The "paths" might be a bit confusing for people not familiar with that parquet terminology. Column "names" ?

[wesm]

Hm. I don't think you can use Parquet and hide from this detail. To give an example of what I mean, you have to say field_name.list.item to refer to the inner column for a type like list<string>. I'm open to improving the usability of this but I don't want to spend a lot of energy on it while we have the Datasets C++ project pending in the near future

[jorisvandenbossche]

I think an example explaining this in the parquet section would already clarify a lot (and enough, I didn't meant to suggest to change the API itself).

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

OK. I'll expand the docstring and give a couple examples

[jorisvandenbossche]

Does this already work for a partitioned dataset with multiple parquet files where a the different files might have different set of unique values?

[wesm]

Each chunk in the table will have a different dictionary, yeah. So there shouldn't be any problem

[wesm]

I'll expand the unit test to check explicitly

[wesm]

Done

[wesm]

I am fine with that, we just need to know that this means we can't really add support for list of rows (there was recently a JIRA about this), as well that it deviates from pandas.DataFrame(..) (which treats lists of arrays as list of rows. But anyway, since Table is a columnar store it totally makes sense to have list of columns as prime use case in pa.table(..)

Yeah I think a list-of-rows should be like a Table.from_records or similar

[wesm]

I'll fix the Windows DLL symbol visibility issues here shortly. @xhochy do you have any opinions about the API?

[xhochy]

+1, I like this API-wise

[xhochy]

Is the limitation intended or simply because we only have it implemented for binary columns?

[wesm]

It's only implemented for BYTE_ARRAY columns at the moment. We could expand that but there is little benefit from a performance/memory use point of view for the primitive types

[xhochy]

I've also used this (through pandas.Categorical) in the past on date and float types (e.g. in some datasets you can have 1000s of products that only have one of 5 prices). This often gave a 4-6x improvement in memory usage for these columns.

[xhochy]

(just dropping it here as FYI)

[wesm]

I see. I'll open a JIRA as a follow up

[wesm]

https://issues.apache.org/jira/browse/ARROW-6140

[xhochy]

column names or paths should be a good alternative that neither hides the format details nor confuses new users.

[wesm]

I just pushed a docstring only fix. Merging this

[wesm]

Oops, my doc fix broke the Python 2.7 build. I will fix