GH-41246: [C++][Python] Simplify nested field encryption configuration

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.

[EnricoMi]

Rationale for this change

Columns can b encrypted with individual keys. For this, the column name have to be set in EncryptionConfiguration::column_keys. This poses the following challenges for columns with nested fields like MapType, ListType, and StructType. Encrypting a column of such type requires providing an encryption key for all nested (leaf) fields. Ideally, the column name should be sufficient (as it is for any other data type) to encrypt all nested fields.

What changes are included in this PR?

The column name can be used to encrypt all nested fields of StrutType, MapType, and ListType columns with the same encryption key. The current column naming scheme can still be used for backward compatibility.

Are these changes tested?

Tested in C++ and Python.

Are there any user-facing changes?

Column encryption can be configured with less code and more intuitive naming.

Documentation and examples updated.

Fixes#41246.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[AlenkaF]

The Python tests LGTM - they are vey concise and reflect the C++ tests.

The C++ addition of FileEncryptionProperties::encrypt_schema looks good to me also but will need a check from a C++ developer. @wgtmac mind having a look?

[AlenkaF]

Note, CI Win failures do not look related but there is a linter fix needed:

@@ -257,11 +257,12 @@ void FileEncryptionProperties::encrypt_schema(const SchemaDescriptor& schema) {
// encrypted_column encrypts column 'it' when 'it' is either equal to
// encrypted_column, or 'it' starts with encrypted_column_prefix,
// i.e. encrypted_column followed by a '.'
- while (it != column_path_vec.end() &&
- (it->first == encrypted_column ||
- // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
- it->first.compare(0, encrypted_column_prefix_len,
- encrypted_column_prefix) == 0)) {
+ while (
+ it != column_path_vec.end() &&
+ (it->first == encrypted_column ||
+ // C++20: can be replaced with it->first.starts_with(encrypted_column_prefix)
+ it->first.compare(0, encrypted_column_prefix_len, encrypted_column_prefix) ==
+ 0)) {

[AlenkaF]

The C++ failure looks related.

[AlenkaF]

cc @pitrou, this is connected to #45411.

[pitrou]

This adds a user-friendly notation for nested fields:

• Columns col.key and col.value can be used to reference they key and value nested field of a MapType column. Currently, col.key_value.key and col.key_value.value are required, respectively.

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively? Otherwise I think we can only keep the following items, which will also simplify the semantics and the implementation:

• Columns col.element can be used to reference they individual list elements of a ListType column. Currently, col.list.element is required.
• The actual column name can be used to encrypt all nested fields with the same encryption key.
• The current column naming scheme can still be used for backward compatibility.

[EnricoMi]

Is this useful? Is there a reasonable use case for using different encryption settings for the map key and value columns, respectively?

Irrespective of it being useful, it is currently possible to do via col.key_value.key and col.key_value.value. This pull request simplifies that notation consistently with lists and structs.

[EnricoMi]

Note that I have removed encrypting the individual list elements via col.element as I cannot see a difference in semantics and encryption output between col.element and col. Both notations are equivalent.

Encrypting a single field of a list of structs column can be denoted as col.f1 rather than col.element.f1.

What do you think?

[EnricoMi]

@pitrou what are your thoughts on the latest logic / semantics?

[EnricoMi]

@pitrou hope you have a moment for this

[pitrou]

I'll be out on vacation, so not before ~10 days I think.

[EnricoMi]

No worries, enjoy!

[pitrou]

@github-actions crossbow submit preview-docs

[pitrou]

Sorry for the delay. This looks in general, a bunch of minor comments and suggestions below.

[github-actions]

Revision: b0ef03f

Submitted crossbow builds: ursacomputing/crossbow @ actions-151fb9c2fa

Task	Status
preview-docs

[pitrou]

Ha, sorry, CI fails now that #48338 was merged, you need to stop passing the column path to ColumnEncryptionProperties::Builder.

[pitrou]

Thanks for the update @EnricoMi !

[pitrou]

I'm sorry that it took so long @EnricoMi . Looking forward to more contributions from you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 052e0aa.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 83 possible false positives for unstable benchmarks that are known to sometimes produce them.