ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 1 commit into from
Closed

ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md #673

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions format/Layout.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
amongst a variety of systems handling flat and nested columnar data, including
such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

|Byte 0 (validity bitmap) | Bytes 1-63 |
|-------------------------|-----------------------|
|00011011 | 0 (padding) |
| 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

|Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
|--------------------------|-------------|-------------|
| 00001011 | 0 (padding) | unspecified |
|Byte 0 (validity bitmap) | Bytes 1-63 |
|-------------------------|-----------------------|
| 00001011 | 0 (padding) |

@wesmwesmMay 11, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:

array->Slice(0, 1)

which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.

Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though

@cloud-fancloud-fanMay 12, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.

However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:

A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).


* Children arrays:
* field-0 array (`List<char>`):
* Length: 4, Null count: 1
* Null bitmap buffer:

| Byte 0 (validity bitmap) | Bytes 1-7 |
| Byte 0 (validity bitmap) | Bytes 1-63 |
|--------------------------|-----------------------|
| 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
to each type. A union with more then 127 possible types can be modeled as a
union of unions.
union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
into the respective child array for the type in a given slot. The respective
offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

|Byte 0 (validity bitmap) | Bytes 1-63 |
|-------------------------|-----------------------|
|00001010 | 0 (padding) |
| 00001010 | 0 (padding) |

* Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

cloud-fan
wants to merge
1
commit into

from

 Closed


ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md
#673
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 11 additions & 11 deletions
format/Layout.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
 amongst a variety of systems handling flat and nested columnar data, including
 such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
 such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
 proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
 linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00011011  | 0 (padding) |
 | 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

 |Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
 |--------------------------|-------------|-------------|
 | 00001011 | 0 (padding) | unspecified |
 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 | 00001011 | 0 (padding)  |
@wesmwesmMay 11, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:
array->Slice(0, 1)
which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.
Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though
@cloud-fancloud-fanMay 12, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.
However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:
A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).

* Children arrays:
 * field-0 array (`List<char>`):
 * Length: 4, Null count: 1
 * Null bitmap buffer:

 | Byte 0 (validity bitmap) | Bytes 1-7  |
 | Byte 0 (validity bitmap) | Bytes 1-63 |
 |--------------------------|-----------------------|
 | 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
 to each type. A union with more then 127 possible types can be modeled as a
 union of unions.
 union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
 into the respective child array for the type in a given slot. The respective
 offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00001010  | 0 (padding) |
 | 00001010 | 0 (padding) |

 * Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

cloud-fan
wants to merge
1
commit into

from

 Closed


ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md
#673
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 11 additions & 11 deletions
format/Layout.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
 amongst a variety of systems handling flat and nested columnar data, including
 such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
 such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
 proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
 linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00011011  | 0 (padding) |
 | 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

 |Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
 |--------------------------|-------------|-------------|
 | 00001011 | 0 (padding) | unspecified |
 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 | 00001011 | 0 (padding)  |
@wesmwesmMay 11, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:
array->Slice(0, 1)
which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.
Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though
@cloud-fancloud-fanMay 12, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.
However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:
A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).

* Children arrays:
 * field-0 array (`List<char>`):
 * Length: 4, Null count: 1
 * Null bitmap buffer:

 | Byte 0 (validity bitmap) | Bytes 1-7  |
 | Byte 0 (validity bitmap) | Bytes 1-63 |
 |--------------------------|-----------------------|
 | 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
 to each type. A union with more then 127 possible types can be modeled as a
 union of unions.
 union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
 into the respective child array for the type in a given slot. The respective
 offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00001010  | 0 (padding) |
 | 00001010 | 0 (padding) |

 * Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

cloud-fan
wants to merge
1
commit into

from

 Closed


ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md
#673
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 11 additions & 11 deletions
format/Layout.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
 amongst a variety of systems handling flat and nested columnar data, including
 such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
 such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
 proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
 linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00011011  | 0 (padding) |
 | 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

 |Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
 |--------------------------|-------------|-------------|
 | 00001011 | 0 (padding) | unspecified |
 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 | 00001011 | 0 (padding)  |
@wesmwesmMay 11, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:
array->Slice(0, 1)
which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.
Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though
@cloud-fancloud-fanMay 12, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.
However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:
A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).

* Children arrays:
 * field-0 array (`List<char>`):
 * Length: 4, Null count: 1
 * Null bitmap buffer:

 | Byte 0 (validity bitmap) | Bytes 1-7  |
 | Byte 0 (validity bitmap) | Bytes 1-63 |
 |--------------------------|-----------------------|
 | 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
 to each type. A union with more then 127 possible types can be modeled as a
 union of unions.
 union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
 into the respective child array for the type in a given slot. The respective
 offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00001010  | 0 (padding) |
 | 00001010 | 0 (padding) |

 * Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

cloud-fan
wants to merge
1
commit into

from

 Closed


ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md
#673
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 11 additions & 11 deletions
format/Layout.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
 amongst a variety of systems handling flat and nested columnar data, including
 such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
 such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
 proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
 linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00011011  | 0 (padding) |
 | 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

 |Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
 |--------------------------|-------------|-------------|
 | 00001011 | 0 (padding) | unspecified |
 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 | 00001011 | 0 (padding)  |
@wesmwesmMay 11, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:
array->Slice(0, 1)
which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.
Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though
@cloud-fancloud-fanMay 12, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.
However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:
A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).

* Children arrays:
 * field-0 array (`List<char>`):
 * Length: 4, Null count: 1
 * Null bitmap buffer:

 | Byte 0 (validity bitmap) | Bytes 1-7  |
 | Byte 0 (validity bitmap) | Bytes 1-63 |
 |--------------------------|-----------------------|
 | 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
 to each type. A union with more then 127 possible types can be modeled as a
 union of unions.
 union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
 into the respective child array for the type in a given slot. The respective
 offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00001010  | 0 (padding) |
 | 00001010 | 0 (padding) |

 * Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

cloud-fan
wants to merge
1
commit into

from

 Closed


ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md
#673
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 11 additions & 11 deletions
format/Layout.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
 amongst a variety of systems handling flat and nested columnar data, including
 such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
 such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
 proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
 linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00011011  | 0 (padding) |
 | 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

 |Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
 |--------------------------|-------------|-------------|
 | 00001011 | 0 (padding) | unspecified |
 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 | 00001011 | 0 (padding)  |
@wesmwesmMay 11, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:
array->Slice(0, 1)
which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.
Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though
@cloud-fancloud-fanMay 12, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.
However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:
A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).

* Children arrays:
 * field-0 array (`List<char>`):
 * Length: 4, Null count: 1
 * Null bitmap buffer:

 | Byte 0 (validity bitmap) | Bytes 1-7  |
 | Byte 0 (validity bitmap) | Bytes 1-63 |
 |--------------------------|-----------------------|
 | 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
 to each type. A union with more then 127 possible types can be modeled as a
 union of unions.
 union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
 into the respective child array for the type in a given slot. The respective
 offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00001010  | 0 (padding) |
 | 00001010 | 0 (padding) |

 * Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

cloud-fan
wants to merge
1
commit into

from

 Closed


ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md
#673
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 11 additions & 11 deletions
format/Layout.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
 amongst a variety of systems handling flat and nested columnar data, including
 such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
 such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
 proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
 linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00011011  | 0 (padding) |
 | 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

 |Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
 |--------------------------|-------------|-------------|
 | 00001011 | 0 (padding) | unspecified |
 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 | 00001011 | 0 (padding)  |
@wesmwesmMay 11, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:
array->Slice(0, 1)
which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.
Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though
@cloud-fancloud-fanMay 12, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.
However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:
A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).

* Children arrays:
 * field-0 array (`List<char>`):
 * Length: 4, Null count: 1
 * Null bitmap buffer:

 | Byte 0 (validity bitmap) | Bytes 1-7  |
 | Byte 0 (validity bitmap) | Bytes 1-63 |
 |--------------------------|-----------------------|
 | 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
 to each type. A union with more then 127 possible types can be modeled as a
 union of unions.
 union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
 into the respective child array for the type in a given slot. The respective
 offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00001010  | 0 (padding) |
 | 00001010 | 0 (padding) |

 * Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md#673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

cloud-fan
wants to merge
1
commit into

from

 Closed


ARROW-1011: [FORMAT] fix typo and mistakes in Layout.md
#673
 Changes from all commits
Commits

Show all changes


1 commit


Select commit


File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


22 changes: 11 additions & 11 deletions
format/Layout.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -51,7 +51,7 @@ Base requirements

* A physical memory layout enabling zero-deserialization data interchange
 amongst a variety of systems handling flat and nested columnar data, including
 such systems as Spark, Drill, Impala, Kudu, Ibis, Spark, ODBC protocols, and
 such systems as Spark, Drill, Impala, Kudu, Ibis, ODBC protocols, and
 proprietary systems that utilize the open source components.
* All array slots are accessible in constant time, with complexity growing
 linearly in the nesting level
Expand DownExpand Up@@ -114,7 +114,7 @@ data-structures over 64 bytes (which will be a common case for Arrow Arrays).

Requiring padding to a multiple of 64 bytes allows for using [SIMD][4] instructions
consistently in loops without additional conditional checks.
This should allow for simpler and more efficient code.
This should allow for simpler and more efficient code.
The specific padding length was chosen because it matches the largest known
SIMD instruction registers available as of April 2016 (Intel AVX-512).
Guaranteed padding can also allow certain compilers
Expand DownExpand Up@@ -146,7 +146,7 @@ signed integer, as it may be as large as the array length.
Any relative type can have null value slots, whether primitive or nested type.

An array with nulls must have a contiguous memory buffer, known as the null (or
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
validity) bitmap, whose length is a multiple of 64 bytes (as discussed above)
and large enough to have at least 1 bit for each array
slot.

Expand DownExpand Up@@ -205,7 +205,7 @@ Would look like:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00011011  | 0 (padding) |
 | 00011011 | 0 (padding) |

* Value Buffer:

Expand DownExpand Up@@ -378,16 +378,16 @@ The layout for [{'joe', 1}, {null, 2}, null, {'mark', 4}] would be:
* Length: 4, Null count: 1
* Null bitmap buffer:

 |Byte 0 (validity bitmap) | Bytes 1-7| Bytes 8-63 |
 |--------------------------|-------------|-------------|
 | 00001011 | 0 (padding) | unspecified |
 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 | 00001011 | 0 (padding)  |
@wesmwesmMay 11, 2017

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC this was a deliberate choice so that one can rely upon accurate null counts using a byte-wise hardware popcount instruction. Though I'm not sure how safe that is in general since we can do:
array->Slice(0, 1)
which would yield a slice of length 1, whose additional bytes should not be inspected in algorithms. In an IPC setting, the padding to byte 63 should be all zeros (see e.g. https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L220), so I think this is fine. We will want to make sure in the IPC writers that we do not write possibly unspecified in-memory bytes from the validity bitmap, instead truncating the buffer to the effective byte range and writing zero padding. It might be worth adding some language to this document about this.
Per our prior discussion about padding -- since this document is about what data gets written to stream or file format, it may be good to be more rigid about zero-ing out the padding bytes that go on the wire. This places a slight additional burden on the writers, though
@cloud-fancloud-fanMay 12, 2017

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we only need to use zero padding for null bitmap in a word(8 bytes) boundary, which means the previous code is accurate.
However, to make the rule simpler, I think it also makes sense to require all the padding bytes to be 0 for null bitmap. This is also what we specified in the "Null bitmaps" section:
A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null. Bitmaps are to be initialized to be all unset at allocation time (this includes padding).

* Children arrays:
 * field-0 array (`List<char>`):
 * Length: 4, Null count: 1
 * Null bitmap buffer:

 | Byte 0 (validity bitmap) | Bytes 1-7  |
 | Byte 0 (validity bitmap) | Bytes 1-63 |
 |--------------------------|-----------------------|
 | 00001101 | 0 (padding) |

Expand DownExpand Up@@ -447,7 +447,7 @@ of overhead for each value. Its physical layout is as follows:
* One child array for each relative type
* Types buffer: A buffer of 8-bit signed integers, enumerated from 0 corresponding
 to each type. A union with more then 127 possible types can be modeled as a
 union of unions.
 union of unions.
* Offsets buffer: A buffer of signed int32 values indicating the relative offset
 into the respective child array for the type in a given slot. The respective
 offsets for each child value array must be in order / increasing.
Expand DownExpand Up@@ -555,7 +555,7 @@ will have the following layout:

 |Byte 0 (validity bitmap) | Bytes 1-63 |
 |-------------------------|-----------------------|
 |00001010  | 0 (padding) |
 | 00001010 | 0 (padding) |

 * Value buffer:

Expand DownExpand Up@@ -636,7 +636,7 @@ type: List<String>

## References

Apache Drill Documentation - [Value Vectors][6]
Apache Drill Documentation - [Value Vectors][6]

[1]: https://en.wikipedia.org/wiki/Bit_numbering
[2]: https://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors
Expand Down