Skip to content

GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jorisvandenbossche merged 8 commits into from
Oct 17, 2023
Merged

GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option #38070

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1161cee
Init doc for encoding
mapleFU Oct 6, 2023
814e041
resolve comment
mapleFU Oct 6, 2023
99d26cf
Merge branch 'main' into parquet-doc/add-doc-for-encoding
mapleFU Oct 6, 2023
6a95840
Merge branch 'main' into parquet-doc/add-doc-for-encoding
mapleFU Oct 9, 2023
bdf82ae
try to fix lint
mapleFU Oct 9, 2023
d38e083
Update python/pyarrow/parquet/core.py
mapleFU Oct 9, 2023
ecec908
add RLE as encoding type
mapleFU Oct 9, 2023
f2869f3
resolve comment and remove 'RLE'
mapleFU Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions python/pyarrow/parquet/core.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
Other features such as compression algorithms or the new serialized
data page format must be enabled separately (see 'compression' and
'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
Specify if we should use dictionary encoding in general or only for
some columns.
compression : str or dict
When encoding the column, if the dictionary size is too large, the
column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
Specify the compression codec, either on a general basis or per-column.
Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
Specify if we should write statistics in general (default is True) or only
for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
and should be combined with a compression codec.
column_encoding : string or dict, default None
Specify the encoding scheme on a per column basis.
Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
Can only be used when when ``use_dictionary`` is set to False, and
cannot be used in combination with ``use_byte_stream_split``.
Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.

@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.

@mapleFUmapleFUOct 6, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...

@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?

@mapleFUmapleFUOct 9, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.

Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.

RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.

@mapleFUmapleFUOct 9, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.

So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this

@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)

Rle is default enabled on Page V2, however, write page v2 is not recommended.

No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)

@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882

@mapleFUmapleFUOct 9, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here

@mapleFUmapleFUOct 10, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.

Certain encodings are only compatible with certain data types.
Please refer to the encodings section of `Reading and writing Parquet
files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option by mapleFU · Pull Request #38070 · apache/arrow · GitHub
Skip to content
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 8 commits into

from

Oct 17, 2023
 Merged


GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option
#38070
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
1161cee

Init doc for encoding


mapleFU Oct 6, 2023
814e041

resolve comment


mapleFU Oct 6, 2023
99d26cf

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 6, 2023
6a95840

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 9, 2023
bdf82ae

try to fix lint


mapleFU Oct 9, 2023
d38e083

Update python/pyarrow/parquet/core.py


mapleFU Oct 9, 2023
ecec908

add RLE as encoding type


mapleFU Oct 9, 2023
f2869f3

resolve comment and remove 'RLE'


mapleFU Oct 10, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


14 changes: 10 additions & 4 deletions
python/pyarrow/parquet/core.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
 Other features such as compression algorithms or the new serialized
 data page format must be enabled separately (see 'compression' and
 'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
 Specify if we should use dictionary encoding in general or only for
 some columns.
compression : str or dict
 When encoding the column, if the dictionary size is too large, the
 column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
 doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
 Specify the compression codec, either on a general basis or per-column.
 Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
 Specify if we should write statistics in general (default is True) or only
 for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
 and should be combined with a compression codec.
column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Can only be used when when ``use_dictionary`` is set to False, and
 cannot be used in combination with ``use_byte_stream_split``.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
 'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.
@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.
@mapleFUmapleFUOct 6, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.
Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.
RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.
So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)
Rle is default enabled on Page V2, however, write page v2 is not recommended.
No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here
@mapleFUmapleFUOct 10, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option by mapleFU · Pull Request #38070 · apache/arrow · GitHub
Skip to content
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 8 commits into

from

Oct 17, 2023
 Merged


GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option
#38070
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
1161cee

Init doc for encoding


mapleFU Oct 6, 2023
814e041

resolve comment


mapleFU Oct 6, 2023
99d26cf

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 6, 2023
6a95840

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 9, 2023
bdf82ae

try to fix lint


mapleFU Oct 9, 2023
d38e083

Update python/pyarrow/parquet/core.py


mapleFU Oct 9, 2023
ecec908

add RLE as encoding type


mapleFU Oct 9, 2023
f2869f3

resolve comment and remove 'RLE'


mapleFU Oct 10, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


14 changes: 10 additions & 4 deletions
python/pyarrow/parquet/core.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
 Other features such as compression algorithms or the new serialized
 data page format must be enabled separately (see 'compression' and
 'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
 Specify if we should use dictionary encoding in general or only for
 some columns.
compression : str or dict
 When encoding the column, if the dictionary size is too large, the
 column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
 doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
 Specify the compression codec, either on a general basis or per-column.
 Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
 Specify if we should write statistics in general (default is True) or only
 for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
 and should be combined with a compression codec.
column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Can only be used when when ``use_dictionary`` is set to False, and
 cannot be used in combination with ``use_byte_stream_split``.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
 'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.
@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.
@mapleFUmapleFUOct 6, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.
Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.
RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.
So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)
Rle is default enabled on Page V2, however, write page v2 is not recommended.
No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here
@mapleFUmapleFUOct 10, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option by mapleFU · Pull Request #38070 · apache/arrow · GitHub
Skip to content
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 8 commits into

from

Oct 17, 2023
 Merged


GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option
#38070
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
1161cee

Init doc for encoding


mapleFU Oct 6, 2023
814e041

resolve comment


mapleFU Oct 6, 2023
99d26cf

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 6, 2023
6a95840

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 9, 2023
bdf82ae

try to fix lint


mapleFU Oct 9, 2023
d38e083

Update python/pyarrow/parquet/core.py


mapleFU Oct 9, 2023
ecec908

add RLE as encoding type


mapleFU Oct 9, 2023
f2869f3

resolve comment and remove 'RLE'


mapleFU Oct 10, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


14 changes: 10 additions & 4 deletions
python/pyarrow/parquet/core.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
 Other features such as compression algorithms or the new serialized
 data page format must be enabled separately (see 'compression' and
 'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
 Specify if we should use dictionary encoding in general or only for
 some columns.
compression : str or dict
 When encoding the column, if the dictionary size is too large, the
 column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
 doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
 Specify the compression codec, either on a general basis or per-column.
 Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
 Specify if we should write statistics in general (default is True) or only
 for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
 and should be combined with a compression codec.
column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Can only be used when when ``use_dictionary`` is set to False, and
 cannot be used in combination with ``use_byte_stream_split``.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
 'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.
@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.
@mapleFUmapleFUOct 6, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.
Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.
RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.
So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)
Rle is default enabled on Page V2, however, write page v2 is not recommended.
No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here
@mapleFUmapleFUOct 10, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option by mapleFU · Pull Request #38070 · apache/arrow · GitHub
Skip to content
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 8 commits into

from

Oct 17, 2023
 Merged


GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option
#38070
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
1161cee

Init doc for encoding


mapleFU Oct 6, 2023
814e041

resolve comment


mapleFU Oct 6, 2023
99d26cf

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 6, 2023
6a95840

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 9, 2023
bdf82ae

try to fix lint


mapleFU Oct 9, 2023
d38e083

Update python/pyarrow/parquet/core.py


mapleFU Oct 9, 2023
ecec908

add RLE as encoding type


mapleFU Oct 9, 2023
f2869f3

resolve comment and remove 'RLE'


mapleFU Oct 10, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


14 changes: 10 additions & 4 deletions
python/pyarrow/parquet/core.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
 Other features such as compression algorithms or the new serialized
 data page format must be enabled separately (see 'compression' and
 'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
 Specify if we should use dictionary encoding in general or only for
 some columns.
compression : str or dict
 When encoding the column, if the dictionary size is too large, the
 column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
 doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
 Specify the compression codec, either on a general basis or per-column.
 Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
 Specify if we should write statistics in general (default is True) or only
 for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
 and should be combined with a compression codec.
column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Can only be used when when ``use_dictionary`` is set to False, and
 cannot be used in combination with ``use_byte_stream_split``.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
 'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.
@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.
@mapleFUmapleFUOct 6, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.
Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.
RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.
So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)
Rle is default enabled on Page V2, however, write page v2 is not recommended.
No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here
@mapleFUmapleFUOct 10, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option by mapleFU · Pull Request #38070 · apache/arrow · GitHub
Skip to content
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 8 commits into

from

Oct 17, 2023
 Merged


GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option
#38070
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
1161cee

Init doc for encoding


mapleFU Oct 6, 2023
814e041

resolve comment


mapleFU Oct 6, 2023
99d26cf

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 6, 2023
6a95840

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 9, 2023
bdf82ae

try to fix lint


mapleFU Oct 9, 2023
d38e083

Update python/pyarrow/parquet/core.py


mapleFU Oct 9, 2023
ecec908

add RLE as encoding type


mapleFU Oct 9, 2023
f2869f3

resolve comment and remove 'RLE'


mapleFU Oct 10, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


14 changes: 10 additions & 4 deletions
python/pyarrow/parquet/core.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
 Other features such as compression algorithms or the new serialized
 data page format must be enabled separately (see 'compression' and
 'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
 Specify if we should use dictionary encoding in general or only for
 some columns.
compression : str or dict
 When encoding the column, if the dictionary size is too large, the
 column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
 doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
 Specify the compression codec, either on a general basis or per-column.
 Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
 Specify if we should write statistics in general (default is True) or only
 for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
 and should be combined with a compression codec.
column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Can only be used when when ``use_dictionary`` is set to False, and
 cannot be used in combination with ``use_byte_stream_split``.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
 'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.
@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.
@mapleFUmapleFUOct 6, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.
Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.
RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.
So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)
Rle is default enabled on Page V2, however, write page v2 is not recommended.
No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here
@mapleFUmapleFUOct 10, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option by mapleFU · Pull Request #38070 · apache/arrow · GitHub
Skip to content
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 8 commits into

from

Oct 17, 2023
 Merged


GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option
#38070
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
1161cee

Init doc for encoding


mapleFU Oct 6, 2023
814e041

resolve comment


mapleFU Oct 6, 2023
99d26cf

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 6, 2023
6a95840

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 9, 2023
bdf82ae

try to fix lint


mapleFU Oct 9, 2023
d38e083

Update python/pyarrow/parquet/core.py


mapleFU Oct 9, 2023
ecec908

add RLE as encoding type


mapleFU Oct 9, 2023
f2869f3

resolve comment and remove 'RLE'


mapleFU Oct 10, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


14 changes: 10 additions & 4 deletions
python/pyarrow/parquet/core.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
 Other features such as compression algorithms or the new serialized
 data page format must be enabled separately (see 'compression' and
 'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
 Specify if we should use dictionary encoding in general or only for
 some columns.
compression : str or dict
 When encoding the column, if the dictionary size is too large, the
 column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
 doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
 Specify the compression codec, either on a general basis or per-column.
 Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
 Specify if we should write statistics in general (default is True) or only
 for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
 and should be combined with a compression codec.
column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Can only be used when when ``use_dictionary`` is set to False, and
 cannot be used in combination with ``use_byte_stream_split``.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
 'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.
@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.
@mapleFUmapleFUOct 6, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.
Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.
RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.
So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)
Rle is default enabled on Page V2, however, write page v2 is not recommended.
No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here
@mapleFUmapleFUOct 10, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option by mapleFU · Pull Request #38070 · apache/arrow · GitHub
Skip to content
GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option#38070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

jorisvandenbossche
merged 8 commits into

from

Oct 17, 2023
 Merged


GH-37312: [Python][Docs] Update Python docstrings to reflect new parquet encoding option
#38070
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
1161cee

Init doc for encoding


mapleFU Oct 6, 2023
814e041

resolve comment


mapleFU Oct 6, 2023
99d26cf

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 6, 2023
6a95840

Merge branch 'main' into parquet-doc/add-doc-for-encoding


mapleFU Oct 9, 2023
bdf82ae

try to fix lint


mapleFU Oct 9, 2023
d38e083

Update python/pyarrow/parquet/core.py


mapleFU Oct 9, 2023
ecec908

add RLE as encoding type


mapleFU Oct 9, 2023
f2869f3

resolve comment and remove 'RLE'


mapleFU Oct 10, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


14 changes: 10 additions & 4 deletions
python/pyarrow/parquet/core.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -767,13 +767,16 @@ def _sanitize_table(table, new_schema, flavor):
 Other features such as compression algorithms or the new serialized
 data page format must be enabled separately (see 'compression' and
 'data_page_version').
use_dictionary : bool or list
use_dictionary : bool or list, default True
 Specify if we should use dictionary encoding in general or only for
 some columns.
compression : str or dict
 When encoding the column, if the dictionary size is too large, the
 column will fallback to ``PLAIN`` encoding. Specially, ``BOOLEAN`` type
 doesn't support dictionary encoding.
compression : str or dict, default 'snappy'
 Specify the compression codec, either on a general basis or per-column.
 Valid values: {'NONE', 'SNAPPY', 'GZIP', 'BROTLI', 'LZ4', 'ZSTD'}.
write_statistics : bool or list
write_statistics : bool or list, default True
 Specify if we should write statistics in general (default is True) or only
 for some columns.
use_deprecated_int96_timestamps : bool, default None
Expand DownExpand Up@@ -821,7 +824,10 @@ def _sanitize_table(table, new_schema, flavor):
 and should be combined with a compression codec.
column_encoding : string or dict, default None
 Specify the encoding scheme on a per column basis.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}.
 Can only be used when when ``use_dictionary`` is set to False, and
 cannot be used in combination with ``use_byte_stream_split``.
 Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT',
 'DELTA_BINARY_PACKED', 'DELTA_LENGTH_BYTE_ARRAY', 'DELTA_BYTE_ARRAY'}.
@jorisvandenbosschejorisvandenbosscheOct 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, "RLE" is also allowed, but since that is now the default and only valid for booleans, there is probably no use in actually specifying it.
@mapleFUmapleFUOct 6, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔As we mentioned in https://issues.apache.org/jira/browse/PARQUET-2222 . I think export RLE is not a so good idea...
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But so then we should revert #36955, since that enabled RLE by default for boolean columns?
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rle is default enabled on Page V2, however, write page v2 is not recommended.
Page V2 format is great, however, most parquet implementions didn't get agreement on it. As a result, page v2 is unstable now.
RLE Boolean is default enabled on page v2, I think it's ok there, but I don't think is a good idea to default enable it.
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as this https://blog.getdaft.io/p/working-with-the-apache-parquet-file blog saying. Parquet V2 is a ambigious naming. Although we (arrow and arrow-rs ) is using format 2.x and some properties on it. Most of implementions can still decode page v1.
So I think if user know what he/she is doing, RLE is ok to export to user, however I think here we can just hide it until PARQUET-2222 has a conclusion about this
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to move this discussion elsewhere (new issue? or the original issue of #36882)
Rle is default enabled on Page V2, however, write page v2 is not recommended.
No, we enabled it by default also for DataPage V1 (we have a separate "parquet v2" which is enabled by default, and we decided to write this encoding when that is enabled)
@jorisvandenbosschejorisvandenbosscheOct 9, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to #36882
@mapleFUmapleFUOct 9, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I've added RLE here
@mapleFUmapleFUOct 10, 2023

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh finally I decide to remove RLE. Just leave it as a hack, until we make clear how it could be used.
 Certain encodings are only compatible with certain data types.
 Please refer to the encodings section of `Reading and writing Parquet
 files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.
Expand Down