GH-41246: [Docs][C++][Python] Improve docs on column encryption for nested fields

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[EnricoMi]

Rationale for this change

Encrypting columns with nested fields with a column key is not trivial since only leaf fields are allowed in the column key map. Documentation emphasizes this fact and provides examples.

What changes are included in this PR?

This amends the documentation on encryption for C++ and Python.

Are these changes tested?

Only documentation.

Are there any user-facing changes?

Only documentation.

• GitHub Issue: [Python][Parquet] Attempt to encrypt column of type 'list' produces OSError #41246

[github-actions]

⚠️ GitHub issue #41246has been automatically assigned in GitHub to PR creator.

[pitrou]

Thanks for noticing and document this @EnricoMi . Here are assorted comments.

[pitrou]

Are the spaces embedded in the string actually supported? Also, it seems to lack a semicolon at the end of the line.

[EnricoMi]

Yes it supports extra whitespaces, it trims the strings. The final semicolon is not needed, it would introduce a next empty section:

arrow/cpp/src/parquet/encryption/crypto_factory.cc

Lines 101 to 167 in 240ebc9

	ColumnPathToEncryptionPropertiesMap CryptoFactory::GetColumnEncryptionProperties(
	int dek_length, const std::string& column_keys, FileKeyWrapper* key_wrapper) {
	ColumnPathToEncryptionPropertiesMap encrypted_columns;
	std::vector<::std::string_view> key_to_columns =
	::arrow::internal::SplitString(column_keys, ';');
	for (size_t i = 0; i < key_to_columns.size(); ++i) {
	std::string cur_key_to_columns =
	::arrow::internal::TrimString(std::string(key_to_columns[i]));
	if (cur_key_to_columns.empty()) {
	continue;
	}
	std::vector<::std::string_view> parts =
	::arrow::internal::SplitString(cur_key_to_columns, ':');
	if (parts.size() != 2) {
	std::ostringstream message;
	message << "Incorrect key to columns mapping in column keys property"
	<< ": [" << cur_key_to_columns << "]";
	throwParquetException(message.str());
	}
	std::string column_key_id = ::arrow::internal::TrimString(std::string(parts[0]));
	if (column_key_id.empty()) {
	throwParquetException("Empty key name in column keys property.");
	}
	std::string column_names_str = ::arrow::internal::TrimString(std::string(parts[1]));
	std::vector<::std::string_view> column_names =
	::arrow::internal::SplitString(column_names_str, ',');
	if (0 == column_names.size()) {
	throwParquetException("No columns to encrypt defined for key: " + column_key_id);
	}
	for (size_t j = 0; j < column_names.size(); ++j) {
	std::string column_name =
	::arrow::internal::TrimString(std::string(column_names[j]));
	if (column_name.empty()) {
	std::ostringstream message;
	message << "Empty column name in column keys property for key: " << column_key_id;
	throwParquetException(message.str());
	}
	if (encrypted_columns.find(column_name) != encrypted_columns.end()) {
	throwParquetException("Multiple keys defined for the same column: " +
	column_name);
	}
	std::string column_key(dek_length, '\0');
	RandBytes(reinterpret_cast<uint8_t*>(column_key.data()), column_key.size());
	std::string column_key_key_metadata =
	key_wrapper->GetEncryptionKeyMetadata(column_key, column_key_id, false);
	std::shared_ptr<ColumnEncryptionProperties> cmd =
	ColumnEncryptionProperties::Builder(column_name)
	.key(column_key)
	->key_metadata(column_key_key_metadata)
	->build();
	encrypted_columns.insert({column_name, cmd});
	}
	}
	if (encrypted_columns.empty()) {
	throwParquetException("No column keys configured in column keys property.");
	}
	return encrypted_columns;
	}

[pitrou]

@jorisvandenbossche@AlenkaF@raulcd What is our preferred policy for code examples? Do we put them inline in the docs? Do we use separate files?

[pitrou]

Update: it seems we use literalinclude directives from C++ example files that are compiled as part of CI runs. See for example https://github.com/apache/arrow/blob/main/docs/source/cpp/dataset.rst#reading-datasets

[EnricoMi]

There is some improvement for this non-intuitive naming scheme: #45462

[pitrou]

This LGTM except that we probably want to live in a separate file (see comment below).

[EnricoMi]

@pitrou moved the C++ code into a file and referenced that from parquet.rst.

[EnricoMi]

@pitrou addressed your comments, can this docs improvement be merged?

[pitrou]

The CI failures seem unexpected, can you rebase/merge from latest git main?

[EnricoMi]

Rebased with latest main commit d88ef57.

[pitrou]

Ok, the CI failures are certainly unrelated.

[pitrou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/13717903134

[AlenkaF]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 52fb219

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c034c02a1

Task	Status
preview-docs

[AlenkaF]

Thank you for the updates! Here are the rendered files:

• http://crossbow.voltrondata.com/pr_docs/45411/cpp/parquet.html#encryption
• http://crossbow.voltrondata.com/pr_docs/45411/cpp/examples/parquet_column_encryption.html
• http://crossbow.voltrondata.com/pr_docs/45411/python/parquet.html#encryption-configuration

Looks good to me. Also the CI failures are not related.

[EnricoMi]

@pitrou are you happy with this?

[pitrou]

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map or list data types)
	requires column keys for the inner fields, not the nested column itself.
	Configuring a column key for the nested column causes
	this error (here the column name is ``col``):

[EnricoMi]

I find nested column ambiguous, it sounds like the inner column, the nested field.

What about outer column instead of nested column?

Suggested change

	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the column itself.
	Configuring a column key for the column itself causes this error (here column name is ``col``):
	Encrypting columns that have nested fields (struct, map, or even list data types)
	requires column keys for the inner fields, not the outer column itself.
	Configuring a column key for the outer column causes this error (here column name is ``col``):

[pitrou]

You're right, that's better!

[pitrou]

Also two wording nits:

• "struct, map, or even list data types" -> "struct, map, or list data types" (there's nothing special about lists)
• "here column name is" -> "here the column name is"

[EnricoMi]

Sorry, missed those of your changes when creating my suggestion. All incorporated now.

[pitrou]

(same suggestions as for C++)

[pitrou]

IMHO We should avoid using this header in public examples. std::cerr is good enough here.

[pitrou]

Can you move this include together with other Arrow includes above?

[EnricoMi]

@pitrou all comments addressed, one alternative suggested at #45411 (comment)

[pitrou]

Small nit

Suggested change

	#include"arrow/filesystem//localfs.h"
	#include"arrow/filesystem/localfs.h"

[EnricoMi]

@pitrou comments addressed

[pitrou]

Thanks a lot for doing this @EnricoMi !

[pitrou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: b47168d

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea617ad0a3

Task	Status
preview-docs

[EnricoMi]

Thank you for your time, @pitrou!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 5d0149d.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.