ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
emkornfield wants to merge 6 commits into from
Closed

ARROW-12110: [Java] Implement ZSTD compression #9822

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b5bcf68
ARROW-12110: [Java] Implement ZSTD compression
emkornfield Mar 27, 2021
c324527
add missing file
emkornfield Mar 27, 2021
b97e83f
Update runner.py
Mar 27, 2021
cd0f4ba
Update ZstdCompressionCodec.java
Mar 27, 2021
c9216a8
Update runner.py
Mar 27, 2021
944f655
Merge branch 'master' into zstd
Mar 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions dev/archery/archery/integration/runner.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
skip.add("Go")
skip.add("JS")
skip.add("Rust")
if name == 'zstd':

@nevi-menevi-meMar 27, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).

Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?

@emkornfieldemkornfieldMar 27, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>

skip.add("Java")

# See https://github.com/apache/arrow/pull/9822 for how to
# disable specific compression type tests.

if prefix == '4.0.0-shareddict':
skip.add("Go")

yield datagen.File(name, None, None, skip=skip, path=out_path)

def _run_test_cases(self, producer, consumer, case_runner,
Expand Down
8 changes: 6 additions & 2 deletions docs/source/status.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ | | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W | | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.
7 changes: 4 additions & 3 deletions java/compression/pom.xml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
<version>1.20</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-common</artifactId>
</dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>1.4.9-1</version>
</dependency>
</dependencies>
</project>
6 changes: 5 additions & 1 deletion java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
* A factory implementation based on Apache Commons library.
* Default implementation of factory supported LZ4 and ZSTD compression.
*
* // TODO(ARROW-12115): Rename this class.
*/
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
switch (codecType) {
case LZ4_FRAME:
return new Lz4CompressionCodec();
case ZSTD:
return new ZstdCompressionCodec();
default:
throw new IllegalArgumentException("Compression type not supported: " + codecType);
}
Expand Down
13 changes: 4 additions & 9 deletions java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
* Compression codec for the LZ4 algorithm.
*/
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
"The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
uncompressedBuffer.getBytes(/*index=*/0, inBytes);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (InputStream in = new ByteArrayInputStream(inBytes);
OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
byte[] outBytes = baos.toByteArray();

ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
PlatformDependent.copyMemory(
outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
return compressedBuffer;
}
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
long decompressedLength = readUncompressedLength(compressedBuffer);

byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
PlatformDependent.copyMemory(
compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

byte[] outBytes = out.toByteArray();
ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
decompressedBuffer.writerIndex(decompressedLength);
decompressedBuffer.setBytes(/*index=*/0, outBytes);
return decompressedBuffer;
}

Expand Down
74 changes: 74 additions & 0 deletions java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
* Compression codec for the ZSTD algorithm.
*/
public class ZstdCompressionCodec extends AbstractCompressionCodec {

@Override
protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());

@liyafan82liyafan82Mar 29, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?

@emkornfieldemkornfieldMar 29, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.

long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
ArrowBuf compressedBuffer = allocator.buffer(dstSize);
long bytesWritten = Zstd.compressUnsafe(
compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
/*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
/*level=*/3);

@liyafan82liyafan82Mar 29, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?

@emkornfieldemkornfieldMar 29, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.

@emkornfieldemkornfieldMar 29, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.

if (Zstd.isError(bytesWritten)) {
compressedBuffer.close();
throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
}
compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
return compressedBuffer;
}

@Override
protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
long decompressedLength = readUncompressedLength(compressedBuffer);
ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
/*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
if (Zstd.isError(decompressedSize)) {
uncompressedBuffer.close();
throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
}
if (decompressedLength != decompressedSize) {
uncompressedBuffer.close();
throw new RuntimeException("Expected != actual decompressed length: " +
decompressedLength + " != " + decompressedSize);
}
uncompressedBuffer.writerIndex(decompressedLength);
return uncompressedBuffer;
}

@Override
public CompressionUtil.CodecType getCodecType() {
return CompressionUtil.CodecType.ZSTD;
}
}
4 changes: 4 additions & 0 deletions java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

CompressionCodec lz4Codec = new Lz4CompressionCodec();
params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

CompressionCodec zstdCodec = new ZstdCompressionCodec();
params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

}
return params;
}
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
6
commits into

from

 Closed


ARROW-12110: [Java] Implement ZSTD compression
#9822
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b5bcf68

ARROW-12110: [Java] Implement ZSTD compression


emkornfield Mar 27, 2021
c324527

add missing file


emkornfield Mar 27, 2021
b97e83f

Update runner.py

Mar 27, 2021
cd0f4ba

Update ZstdCompressionCodec.java

Mar 27, 2021
c9216a8

Update runner.py

Mar 27, 2021
944f655

Merge branch 'master' into zstd

Mar 29, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


7 changes: 5 additions & 2 deletions
dev/archery/archery/integration/runner.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
 skip.add("Go")
 skip.add("JS")
 skip.add("Rust")
 if name == 'zstd':
@nevi-menevi-meMar 27, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).
Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?
@emkornfieldemkornfieldMar 27, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>
 skip.add("Java")

 # See https://github.com/apache/arrow/pull/9822 for how to
 # disable specific compression type tests.

 if prefix == '4.0.0-shareddict':
 skip.add("Go")

 yield datagen.File(name, None, None, skip=skip, path=out_path)

 def _run_test_cases(self, producer, consumer, case_runner,
Expand Down

8 changes: 6 additions & 2 deletions
docs/source/status.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ |  | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
 The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W |  | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.

7 changes: 4 additions & 3 deletions
java/compression/pom.xml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
 <version>1.20</version>
 </dependency>
 <dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-common</artifactId>
 </dependency>
 <groupId>com.github.luben</groupId>
 <artifactId>zstd-jni</artifactId>
 <version>1.4.9-1</version>
</dependency>
 </dependencies>
</project>

6 changes: 5 additions & 1 deletion
java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
 * A factory implementation based on Apache Commons library.
 * Default implementation of factory supported LZ4 and ZSTD compression.
 *
 * // TODO(ARROW-12115): Rename this class.
 */
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
 switch (codecType) {
 case LZ4_FRAME:
 return new Lz4CompressionCodec();
 case ZSTD:
 return new ZstdCompressionCodec();
 default:
 throw new IllegalArgumentException("Compression type not supported: " + codecType);
 }
Expand Down

13 changes: 4 additions & 9 deletions
java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
 * Compression codec for the LZ4 algorithm.
 */
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 "The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

 byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
 uncompressedBuffer.getBytes(/*index=*/0, inBytes);
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 try (InputStream in = new ByteArrayInputStream(inBytes);
 OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 byte[] outBytes = baos.toByteArray();

 ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 PlatformDependent.copyMemory(
 outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
 compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 return compressedBuffer;
 }
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
 long decompressedLength = readUncompressedLength(compressedBuffer);

 byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
 PlatformDependent.copyMemory(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
 compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
 ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
 try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
 IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

 byte[] outBytes = out.toByteArray();
 ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
 PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
 decompressedBuffer.writerIndex(decompressedLength);
 decompressedBuffer.setBytes(/*index=*/0, outBytes);
 return decompressedBuffer;
 }

Expand Down

74 changes: 74 additions & 0 deletions
java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
 * Compression codec for the ZSTD algorithm.
 */
public class ZstdCompressionCodec extends AbstractCompressionCodec {

 @Override
 protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
 long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.
 long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
 ArrowBuf compressedBuffer = allocator.buffer(dstSize);
 long bytesWritten = Zstd.compressUnsafe(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
 /*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
 /*level=*/3);
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.
 if (Zstd.isError(bytesWritten)) {
 compressedBuffer.close();
 throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
 }
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
 return compressedBuffer;
 }

 @Override
 protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
 long decompressedLength = readUncompressedLength(compressedBuffer);
 ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
 long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
 /*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
 compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
 if (Zstd.isError(decompressedSize)) {
 uncompressedBuffer.close();
 throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
 }
 if (decompressedLength != decompressedSize) {
 uncompressedBuffer.close();
 throw new RuntimeException("Expected != actual decompressed length: " + 
 decompressedLength + " != " + decompressedSize);
 }
 uncompressedBuffer.writerIndex(decompressedLength);
 return uncompressedBuffer;
 }

 @Override
 public CompressionUtil.CodecType getCodecType() {
 return CompressionUtil.CodecType.ZSTD;
 }
}

4 changes: 4 additions & 0 deletions
java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

 CompressionCodec lz4Codec = new Lz4CompressionCodec();
 params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

 CompressionCodec zstdCodec = new ZstdCompressionCodec();
 params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

 }
 return params;
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
6
commits into

from

 Closed


ARROW-12110: [Java] Implement ZSTD compression
#9822
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b5bcf68

ARROW-12110: [Java] Implement ZSTD compression


emkornfield Mar 27, 2021
c324527

add missing file


emkornfield Mar 27, 2021
b97e83f

Update runner.py

Mar 27, 2021
cd0f4ba

Update ZstdCompressionCodec.java

Mar 27, 2021
c9216a8

Update runner.py

Mar 27, 2021
944f655

Merge branch 'master' into zstd

Mar 29, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


7 changes: 5 additions & 2 deletions
dev/archery/archery/integration/runner.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
 skip.add("Go")
 skip.add("JS")
 skip.add("Rust")
 if name == 'zstd':
@nevi-menevi-meMar 27, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).
Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?
@emkornfieldemkornfieldMar 27, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>
 skip.add("Java")

 # See https://github.com/apache/arrow/pull/9822 for how to
 # disable specific compression type tests.

 if prefix == '4.0.0-shareddict':
 skip.add("Go")

 yield datagen.File(name, None, None, skip=skip, path=out_path)

 def _run_test_cases(self, producer, consumer, case_runner,
Expand Down

8 changes: 6 additions & 2 deletions
docs/source/status.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ |  | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
 The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W |  | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.

7 changes: 4 additions & 3 deletions
java/compression/pom.xml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
 <version>1.20</version>
 </dependency>
 <dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-common</artifactId>
 </dependency>
 <groupId>com.github.luben</groupId>
 <artifactId>zstd-jni</artifactId>
 <version>1.4.9-1</version>
</dependency>
 </dependencies>
</project>

6 changes: 5 additions & 1 deletion
java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
 * A factory implementation based on Apache Commons library.
 * Default implementation of factory supported LZ4 and ZSTD compression.
 *
 * // TODO(ARROW-12115): Rename this class.
 */
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
 switch (codecType) {
 case LZ4_FRAME:
 return new Lz4CompressionCodec();
 case ZSTD:
 return new ZstdCompressionCodec();
 default:
 throw new IllegalArgumentException("Compression type not supported: " + codecType);
 }
Expand Down

13 changes: 4 additions & 9 deletions
java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
 * Compression codec for the LZ4 algorithm.
 */
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 "The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

 byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
 uncompressedBuffer.getBytes(/*index=*/0, inBytes);
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 try (InputStream in = new ByteArrayInputStream(inBytes);
 OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 byte[] outBytes = baos.toByteArray();

 ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 PlatformDependent.copyMemory(
 outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
 compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 return compressedBuffer;
 }
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
 long decompressedLength = readUncompressedLength(compressedBuffer);

 byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
 PlatformDependent.copyMemory(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
 compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
 ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
 try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
 IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

 byte[] outBytes = out.toByteArray();
 ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
 PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
 decompressedBuffer.writerIndex(decompressedLength);
 decompressedBuffer.setBytes(/*index=*/0, outBytes);
 return decompressedBuffer;
 }

Expand Down

74 changes: 74 additions & 0 deletions
java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
 * Compression codec for the ZSTD algorithm.
 */
public class ZstdCompressionCodec extends AbstractCompressionCodec {

 @Override
 protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
 long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.
 long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
 ArrowBuf compressedBuffer = allocator.buffer(dstSize);
 long bytesWritten = Zstd.compressUnsafe(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
 /*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
 /*level=*/3);
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.
 if (Zstd.isError(bytesWritten)) {
 compressedBuffer.close();
 throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
 }
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
 return compressedBuffer;
 }

 @Override
 protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
 long decompressedLength = readUncompressedLength(compressedBuffer);
 ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
 long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
 /*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
 compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
 if (Zstd.isError(decompressedSize)) {
 uncompressedBuffer.close();
 throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
 }
 if (decompressedLength != decompressedSize) {
 uncompressedBuffer.close();
 throw new RuntimeException("Expected != actual decompressed length: " + 
 decompressedLength + " != " + decompressedSize);
 }
 uncompressedBuffer.writerIndex(decompressedLength);
 return uncompressedBuffer;
 }

 @Override
 public CompressionUtil.CodecType getCodecType() {
 return CompressionUtil.CodecType.ZSTD;
 }
}

4 changes: 4 additions & 0 deletions
java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

 CompressionCodec lz4Codec = new Lz4CompressionCodec();
 params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

 CompressionCodec zstdCodec = new ZstdCompressionCodec();
 params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

 }
 return params;
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
6
commits into

from

 Closed


ARROW-12110: [Java] Implement ZSTD compression
#9822
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b5bcf68

ARROW-12110: [Java] Implement ZSTD compression


emkornfield Mar 27, 2021
c324527

add missing file


emkornfield Mar 27, 2021
b97e83f

Update runner.py

Mar 27, 2021
cd0f4ba

Update ZstdCompressionCodec.java

Mar 27, 2021
c9216a8

Update runner.py

Mar 27, 2021
944f655

Merge branch 'master' into zstd

Mar 29, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


7 changes: 5 additions & 2 deletions
dev/archery/archery/integration/runner.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
 skip.add("Go")
 skip.add("JS")
 skip.add("Rust")
 if name == 'zstd':
@nevi-menevi-meMar 27, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).
Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?
@emkornfieldemkornfieldMar 27, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>
 skip.add("Java")

 # See https://github.com/apache/arrow/pull/9822 for how to
 # disable specific compression type tests.

 if prefix == '4.0.0-shareddict':
 skip.add("Go")

 yield datagen.File(name, None, None, skip=skip, path=out_path)

 def _run_test_cases(self, producer, consumer, case_runner,
Expand Down

8 changes: 6 additions & 2 deletions
docs/source/status.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ |  | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
 The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W |  | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.

7 changes: 4 additions & 3 deletions
java/compression/pom.xml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
 <version>1.20</version>
 </dependency>
 <dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-common</artifactId>
 </dependency>
 <groupId>com.github.luben</groupId>
 <artifactId>zstd-jni</artifactId>
 <version>1.4.9-1</version>
</dependency>
 </dependencies>
</project>

6 changes: 5 additions & 1 deletion
java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
 * A factory implementation based on Apache Commons library.
 * Default implementation of factory supported LZ4 and ZSTD compression.
 *
 * // TODO(ARROW-12115): Rename this class.
 */
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
 switch (codecType) {
 case LZ4_FRAME:
 return new Lz4CompressionCodec();
 case ZSTD:
 return new ZstdCompressionCodec();
 default:
 throw new IllegalArgumentException("Compression type not supported: " + codecType);
 }
Expand Down

13 changes: 4 additions & 9 deletions
java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
 * Compression codec for the LZ4 algorithm.
 */
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 "The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

 byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
 uncompressedBuffer.getBytes(/*index=*/0, inBytes);
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 try (InputStream in = new ByteArrayInputStream(inBytes);
 OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 byte[] outBytes = baos.toByteArray();

 ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 PlatformDependent.copyMemory(
 outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
 compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 return compressedBuffer;
 }
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
 long decompressedLength = readUncompressedLength(compressedBuffer);

 byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
 PlatformDependent.copyMemory(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
 compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
 ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
 try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
 IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

 byte[] outBytes = out.toByteArray();
 ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
 PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
 decompressedBuffer.writerIndex(decompressedLength);
 decompressedBuffer.setBytes(/*index=*/0, outBytes);
 return decompressedBuffer;
 }

Expand Down

74 changes: 74 additions & 0 deletions
java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
 * Compression codec for the ZSTD algorithm.
 */
public class ZstdCompressionCodec extends AbstractCompressionCodec {

 @Override
 protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
 long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.
 long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
 ArrowBuf compressedBuffer = allocator.buffer(dstSize);
 long bytesWritten = Zstd.compressUnsafe(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
 /*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
 /*level=*/3);
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.
 if (Zstd.isError(bytesWritten)) {
 compressedBuffer.close();
 throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
 }
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
 return compressedBuffer;
 }

 @Override
 protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
 long decompressedLength = readUncompressedLength(compressedBuffer);
 ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
 long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
 /*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
 compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
 if (Zstd.isError(decompressedSize)) {
 uncompressedBuffer.close();
 throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
 }
 if (decompressedLength != decompressedSize) {
 uncompressedBuffer.close();
 throw new RuntimeException("Expected != actual decompressed length: " + 
 decompressedLength + " != " + decompressedSize);
 }
 uncompressedBuffer.writerIndex(decompressedLength);
 return uncompressedBuffer;
 }

 @Override
 public CompressionUtil.CodecType getCodecType() {
 return CompressionUtil.CodecType.ZSTD;
 }
}

4 changes: 4 additions & 0 deletions
java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

 CompressionCodec lz4Codec = new Lz4CompressionCodec();
 params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

 CompressionCodec zstdCodec = new ZstdCompressionCodec();
 params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

 }
 return params;
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
6
commits into

from

 Closed


ARROW-12110: [Java] Implement ZSTD compression
#9822
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b5bcf68

ARROW-12110: [Java] Implement ZSTD compression


emkornfield Mar 27, 2021
c324527

add missing file


emkornfield Mar 27, 2021
b97e83f

Update runner.py

Mar 27, 2021
cd0f4ba

Update ZstdCompressionCodec.java

Mar 27, 2021
c9216a8

Update runner.py

Mar 27, 2021
944f655

Merge branch 'master' into zstd

Mar 29, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


7 changes: 5 additions & 2 deletions
dev/archery/archery/integration/runner.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
 skip.add("Go")
 skip.add("JS")
 skip.add("Rust")
 if name == 'zstd':
@nevi-menevi-meMar 27, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).
Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?
@emkornfieldemkornfieldMar 27, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>
 skip.add("Java")

 # See https://github.com/apache/arrow/pull/9822 for how to
 # disable specific compression type tests.

 if prefix == '4.0.0-shareddict':
 skip.add("Go")

 yield datagen.File(name, None, None, skip=skip, path=out_path)

 def _run_test_cases(self, producer, consumer, case_runner,
Expand Down

8 changes: 6 additions & 2 deletions
docs/source/status.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ |  | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
 The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W |  | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.

7 changes: 4 additions & 3 deletions
java/compression/pom.xml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
 <version>1.20</version>
 </dependency>
 <dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-common</artifactId>
 </dependency>
 <groupId>com.github.luben</groupId>
 <artifactId>zstd-jni</artifactId>
 <version>1.4.9-1</version>
</dependency>
 </dependencies>
</project>

6 changes: 5 additions & 1 deletion
java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
 * A factory implementation based on Apache Commons library.
 * Default implementation of factory supported LZ4 and ZSTD compression.
 *
 * // TODO(ARROW-12115): Rename this class.
 */
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
 switch (codecType) {
 case LZ4_FRAME:
 return new Lz4CompressionCodec();
 case ZSTD:
 return new ZstdCompressionCodec();
 default:
 throw new IllegalArgumentException("Compression type not supported: " + codecType);
 }
Expand Down

13 changes: 4 additions & 9 deletions
java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
 * Compression codec for the LZ4 algorithm.
 */
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 "The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

 byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
 uncompressedBuffer.getBytes(/*index=*/0, inBytes);
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 try (InputStream in = new ByteArrayInputStream(inBytes);
 OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 byte[] outBytes = baos.toByteArray();

 ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 PlatformDependent.copyMemory(
 outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
 compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 return compressedBuffer;
 }
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
 long decompressedLength = readUncompressedLength(compressedBuffer);

 byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
 PlatformDependent.copyMemory(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
 compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
 ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
 try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
 IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

 byte[] outBytes = out.toByteArray();
 ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
 PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
 decompressedBuffer.writerIndex(decompressedLength);
 decompressedBuffer.setBytes(/*index=*/0, outBytes);
 return decompressedBuffer;
 }

Expand Down

74 changes: 74 additions & 0 deletions
java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
 * Compression codec for the ZSTD algorithm.
 */
public class ZstdCompressionCodec extends AbstractCompressionCodec {

 @Override
 protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
 long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.
 long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
 ArrowBuf compressedBuffer = allocator.buffer(dstSize);
 long bytesWritten = Zstd.compressUnsafe(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
 /*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
 /*level=*/3);
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.
 if (Zstd.isError(bytesWritten)) {
 compressedBuffer.close();
 throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
 }
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
 return compressedBuffer;
 }

 @Override
 protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
 long decompressedLength = readUncompressedLength(compressedBuffer);
 ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
 long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
 /*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
 compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
 if (Zstd.isError(decompressedSize)) {
 uncompressedBuffer.close();
 throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
 }
 if (decompressedLength != decompressedSize) {
 uncompressedBuffer.close();
 throw new RuntimeException("Expected != actual decompressed length: " + 
 decompressedLength + " != " + decompressedSize);
 }
 uncompressedBuffer.writerIndex(decompressedLength);
 return uncompressedBuffer;
 }

 @Override
 public CompressionUtil.CodecType getCodecType() {
 return CompressionUtil.CodecType.ZSTD;
 }
}

4 changes: 4 additions & 0 deletions
java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

 CompressionCodec lz4Codec = new Lz4CompressionCodec();
 params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

 CompressionCodec zstdCodec = new ZstdCompressionCodec();
 params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

 }
 return params;
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
6
commits into

from

 Closed


ARROW-12110: [Java] Implement ZSTD compression
#9822
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b5bcf68

ARROW-12110: [Java] Implement ZSTD compression


emkornfield Mar 27, 2021
c324527

add missing file


emkornfield Mar 27, 2021
b97e83f

Update runner.py

Mar 27, 2021
cd0f4ba

Update ZstdCompressionCodec.java

Mar 27, 2021
c9216a8

Update runner.py

Mar 27, 2021
944f655

Merge branch 'master' into zstd

Mar 29, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


7 changes: 5 additions & 2 deletions
dev/archery/archery/integration/runner.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
 skip.add("Go")
 skip.add("JS")
 skip.add("Rust")
 if name == 'zstd':
@nevi-menevi-meMar 27, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).
Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?
@emkornfieldemkornfieldMar 27, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>
 skip.add("Java")

 # See https://github.com/apache/arrow/pull/9822 for how to
 # disable specific compression type tests.

 if prefix == '4.0.0-shareddict':
 skip.add("Go")

 yield datagen.File(name, None, None, skip=skip, path=out_path)

 def _run_test_cases(self, producer, consumer, case_runner,
Expand Down

8 changes: 6 additions & 2 deletions
docs/source/status.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ |  | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
 The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W |  | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.

7 changes: 4 additions & 3 deletions
java/compression/pom.xml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
 <version>1.20</version>
 </dependency>
 <dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-common</artifactId>
 </dependency>
 <groupId>com.github.luben</groupId>
 <artifactId>zstd-jni</artifactId>
 <version>1.4.9-1</version>
</dependency>
 </dependencies>
</project>

6 changes: 5 additions & 1 deletion
java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
 * A factory implementation based on Apache Commons library.
 * Default implementation of factory supported LZ4 and ZSTD compression.
 *
 * // TODO(ARROW-12115): Rename this class.
 */
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
 switch (codecType) {
 case LZ4_FRAME:
 return new Lz4CompressionCodec();
 case ZSTD:
 return new ZstdCompressionCodec();
 default:
 throw new IllegalArgumentException("Compression type not supported: " + codecType);
 }
Expand Down

13 changes: 4 additions & 9 deletions
java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
 * Compression codec for the LZ4 algorithm.
 */
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 "The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

 byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
 uncompressedBuffer.getBytes(/*index=*/0, inBytes);
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 try (InputStream in = new ByteArrayInputStream(inBytes);
 OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 byte[] outBytes = baos.toByteArray();

 ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 PlatformDependent.copyMemory(
 outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
 compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 return compressedBuffer;
 }
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
 long decompressedLength = readUncompressedLength(compressedBuffer);

 byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
 PlatformDependent.copyMemory(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
 compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
 ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
 try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
 IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

 byte[] outBytes = out.toByteArray();
 ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
 PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
 decompressedBuffer.writerIndex(decompressedLength);
 decompressedBuffer.setBytes(/*index=*/0, outBytes);
 return decompressedBuffer;
 }

Expand Down

74 changes: 74 additions & 0 deletions
java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
 * Compression codec for the ZSTD algorithm.
 */
public class ZstdCompressionCodec extends AbstractCompressionCodec {

 @Override
 protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
 long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.
 long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
 ArrowBuf compressedBuffer = allocator.buffer(dstSize);
 long bytesWritten = Zstd.compressUnsafe(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
 /*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
 /*level=*/3);
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.
 if (Zstd.isError(bytesWritten)) {
 compressedBuffer.close();
 throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
 }
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
 return compressedBuffer;
 }

 @Override
 protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
 long decompressedLength = readUncompressedLength(compressedBuffer);
 ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
 long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
 /*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
 compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
 if (Zstd.isError(decompressedSize)) {
 uncompressedBuffer.close();
 throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
 }
 if (decompressedLength != decompressedSize) {
 uncompressedBuffer.close();
 throw new RuntimeException("Expected != actual decompressed length: " + 
 decompressedLength + " != " + decompressedSize);
 }
 uncompressedBuffer.writerIndex(decompressedLength);
 return uncompressedBuffer;
 }

 @Override
 public CompressionUtil.CodecType getCodecType() {
 return CompressionUtil.CodecType.ZSTD;
 }
}

4 changes: 4 additions & 0 deletions
java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

 CompressionCodec lz4Codec = new Lz4CompressionCodec();
 params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

 CompressionCodec zstdCodec = new ZstdCompressionCodec();
 params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

 }
 return params;
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
6
commits into

from

 Closed


ARROW-12110: [Java] Implement ZSTD compression
#9822
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b5bcf68

ARROW-12110: [Java] Implement ZSTD compression


emkornfield Mar 27, 2021
c324527

add missing file


emkornfield Mar 27, 2021
b97e83f

Update runner.py

Mar 27, 2021
cd0f4ba

Update ZstdCompressionCodec.java

Mar 27, 2021
c9216a8

Update runner.py

Mar 27, 2021
944f655

Merge branch 'master' into zstd

Mar 29, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


7 changes: 5 additions & 2 deletions
dev/archery/archery/integration/runner.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
 skip.add("Go")
 skip.add("JS")
 skip.add("Rust")
 if name == 'zstd':
@nevi-menevi-meMar 27, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).
Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?
@emkornfieldemkornfieldMar 27, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>
 skip.add("Java")

 # See https://github.com/apache/arrow/pull/9822 for how to
 # disable specific compression type tests.

 if prefix == '4.0.0-shareddict':
 skip.add("Go")

 yield datagen.File(name, None, None, skip=skip, path=out_path)

 def _run_test_cases(self, producer, consumer, case_runner,
Expand Down

8 changes: 6 additions & 2 deletions
docs/source/status.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ |  | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
 The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W |  | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.

7 changes: 4 additions & 3 deletions
java/compression/pom.xml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
 <version>1.20</version>
 </dependency>
 <dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-common</artifactId>
 </dependency>
 <groupId>com.github.luben</groupId>
 <artifactId>zstd-jni</artifactId>
 <version>1.4.9-1</version>
</dependency>
 </dependencies>
</project>

6 changes: 5 additions & 1 deletion
java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
 * A factory implementation based on Apache Commons library.
 * Default implementation of factory supported LZ4 and ZSTD compression.
 *
 * // TODO(ARROW-12115): Rename this class.
 */
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
 switch (codecType) {
 case LZ4_FRAME:
 return new Lz4CompressionCodec();
 case ZSTD:
 return new ZstdCompressionCodec();
 default:
 throw new IllegalArgumentException("Compression type not supported: " + codecType);
 }
Expand Down

13 changes: 4 additions & 9 deletions
java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
 * Compression codec for the LZ4 algorithm.
 */
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 "The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

 byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
 uncompressedBuffer.getBytes(/*index=*/0, inBytes);
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 try (InputStream in = new ByteArrayInputStream(inBytes);
 OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 byte[] outBytes = baos.toByteArray();

 ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 PlatformDependent.copyMemory(
 outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
 compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 return compressedBuffer;
 }
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
 long decompressedLength = readUncompressedLength(compressedBuffer);

 byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
 PlatformDependent.copyMemory(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
 compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
 ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
 try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
 IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

 byte[] outBytes = out.toByteArray();
 ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
 PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
 decompressedBuffer.writerIndex(decompressedLength);
 decompressedBuffer.setBytes(/*index=*/0, outBytes);
 return decompressedBuffer;
 }

Expand Down

74 changes: 74 additions & 0 deletions
java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
 * Compression codec for the ZSTD algorithm.
 */
public class ZstdCompressionCodec extends AbstractCompressionCodec {

 @Override
 protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
 long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.
 long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
 ArrowBuf compressedBuffer = allocator.buffer(dstSize);
 long bytesWritten = Zstd.compressUnsafe(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
 /*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
 /*level=*/3);
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.
 if (Zstd.isError(bytesWritten)) {
 compressedBuffer.close();
 throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
 }
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
 return compressedBuffer;
 }

 @Override
 protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
 long decompressedLength = readUncompressedLength(compressedBuffer);
 ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
 long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
 /*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
 compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
 if (Zstd.isError(decompressedSize)) {
 uncompressedBuffer.close();
 throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
 }
 if (decompressedLength != decompressedSize) {
 uncompressedBuffer.close();
 throw new RuntimeException("Expected != actual decompressed length: " + 
 decompressedLength + " != " + decompressedSize);
 }
 uncompressedBuffer.writerIndex(decompressedLength);
 return uncompressedBuffer;
 }

 @Override
 public CompressionUtil.CodecType getCodecType() {
 return CompressionUtil.CodecType.ZSTD;
 }
}

4 changes: 4 additions & 0 deletions
java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

 CompressionCodec lz4Codec = new Lz4CompressionCodec();
 params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

 CompressionCodec zstdCodec = new ZstdCompressionCodec();
 params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

 }
 return params;
 }
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-12110: [Java] Implement ZSTD compression#9822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
6
commits into

from

 Closed


ARROW-12110: [Java] Implement ZSTD compression
#9822
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


6 commits


Select commit
Hold shift + click to select a range
b5bcf68

ARROW-12110: [Java] Implement ZSTD compression


emkornfield Mar 27, 2021
c324527

add missing file


emkornfield Mar 27, 2021
b97e83f

Update runner.py

Mar 27, 2021
cd0f4ba

Update ZstdCompressionCodec.java

Mar 27, 2021
c9216a8

Update runner.py

Mar 27, 2021
944f655

Merge branch 'master' into zstd

Mar 29, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


7 changes: 5 additions & 2 deletions
dev/archery/archery/integration/runner.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,10 +137,13 @@ def _gold_tests(self, gold_dir):
 skip.add("Go")
 skip.add("JS")
 skip.add("Rust")
 if name == 'zstd':
@nevi-menevi-meMar 27, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside to removing this comment completely is that someone might not easily figure out how to skip tests based on name (when working on another implementation).
Do you mind adding a comment to the effect that if an implementer is skipping one of the compression formats, they could use if name == 'format'?
@emkornfieldemkornfieldMar 27, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a link back to this PR>
 skip.add("Java")

 # See https://github.com/apache/arrow/pull/9822 for how to
 # disable specific compression type tests.

 if prefix == '4.0.0-shareddict':
 skip.add("Go")

 yield datagen.File(name, None, None, skip=skip, path=out_path)

 def _run_test_cases(self, producer, consumer, case_runner,
Expand Down

8 changes: 6 additions & 2 deletions
docs/source/status.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -126,7 +126,7 @@ IPC Format
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Sparse tensors | ✓ | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Buffer compression | ✓ |  | | | | | ✓ |
| Buffer compression | ✓ | ✓ (3) | | | | | ✓ |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
| Endianness conversion | ✓ (2) | | | | | | |
+-----------------------------+-------+-------+-------+------------+-------+-------+-------+
Expand All@@ -139,6 +139,8 @@ Notes:

* \(2) Data with non-native endianness can be byte-swapped automatically when reading.

* \(3) LZ4 Codec currently is quite inefficient. ARROW-11901 tracks improving performance.

.. seealso::
 The :ref:`format-ipc` specification.

Expand DownExpand Up@@ -221,7 +223,7 @@ Third-Party Data Formats
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| ORC | R | | | | | | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+
| Parquet | R/W |  | | | | R/W (1) | |
| Parquet | R/W | R (2) | | | | R/W (1) | |
+-----------------------------+---------+---------+-------+------------+-------+---------+-------+

Notes:
Expand All@@ -231,3 +233,5 @@ Notes:
* *W* = Write supported

* \(1) Nested read/write not supported

* \(2) Through JNI bindings to datasets.

7 changes: 4 additions & 3 deletions
java/compression/pom.xml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -44,8 +44,9 @@
 <version>1.20</version>
 </dependency>
 <dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-common</artifactId>
 </dependency>
 <groupId>com.github.luben</groupId>
 <artifactId>zstd-jni</artifactId>
 <version>1.4.9-1</version>
</dependency>
 </dependencies>
</project>

6 changes: 5 additions & 1 deletion
java/compression/src/main/java/org/apache/arrow/compression/CommonsCompressionFactory.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -21,7 +21,9 @@
import org.apache.arrow.vector.compression.CompressionUtil;

/**
 * A factory implementation based on Apache Commons library.
 * Default implementation of factory supported LZ4 and ZSTD compression.
 *
 * // TODO(ARROW-12115): Rename this class.
 */
public class CommonsCompressionFactory implements CompressionCodec.Factory {

Expand All@@ -32,6 +34,8 @@ public CompressionCodec createCodec(CompressionUtil.CodecType codecType) {
 switch (codecType) {
 case LZ4_FRAME:
 return new Lz4CompressionCodec();
 case ZSTD:
 return new ZstdCompressionCodec();
 default:
 throw new IllegalArgumentException("Compression type not supported: " + codecType);
 }
Expand Down

13 changes: 4 additions & 9 deletions
java/compression/src/main/java/org/apache/arrow/compression/Lz4CompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -32,8 +32,6 @@
import org.apache.commons.compress.compressors.lz4.FramedLZ4CompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;

import io.netty.util.internal.PlatformDependent;

/**
 * Compression codec for the LZ4 algorithm.
 */
Expand All@@ -45,7 +43,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 "The uncompressed buffer size exceeds the integer limit %s.", Integer.MAX_VALUE);

 byte[] inBytes = new byte[(int) uncompressedBuffer.writerIndex()];
PlatformDependent.copyMemory(uncompressedBuffer.memoryAddress(), inBytes, 0, uncompressedBuffer.writerIndex());
 uncompressedBuffer.getBytes(/*index=*/0, inBytes);
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 try (InputStream in = new ByteArrayInputStream(inBytes);
 OutputStream out = new FramedLZ4CompressorOutputStream(baos)) {
Expand All@@ -57,8 +55,7 @@ protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBu
 byte[] outBytes = baos.toByteArray();

 ArrowBuf compressedBuffer = allocator.buffer(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 PlatformDependent.copyMemory(
 outBytes, 0, compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes.length);
 compressedBuffer.setBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, outBytes);
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + outBytes.length);
 return compressedBuffer;
 }
Expand All@@ -71,8 +68,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu
 long decompressedLength = readUncompressedLength(compressedBuffer);

 byte[] inBytes = new byte[(int) (compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH)];
 PlatformDependent.copyMemory(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes, 0, inBytes.length);
 compressedBuffer.getBytes(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, inBytes);
 ByteArrayOutputStream out = new ByteArrayOutputStream((int) decompressedLength);
 try (InputStream in = new FramedLZ4CompressorInputStream(new ByteArrayInputStream(inBytes))) {
 IOUtils.copy(in, out);
Expand All@@ -82,8 +78,7 @@ protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBu

 byte[] outBytes = out.toByteArray();
 ArrowBuf decompressedBuffer = allocator.buffer(outBytes.length);
 PlatformDependent.copyMemory(outBytes, 0, decompressedBuffer.memoryAddress(), outBytes.length);
 decompressedBuffer.writerIndex(decompressedLength);
 decompressedBuffer.setBytes(/*index=*/0, outBytes);
 return decompressedBuffer;
 }

Expand Down

74 changes: 74 additions & 0 deletions
java/compression/src/main/java/org/apache/arrow/compression/ZstdCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.arrow.compression;


import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.vector.compression.AbstractCompressionCodec;
import org.apache.arrow.vector.compression.CompressionUtil;

import com.github.luben.zstd.Zstd;

/**
 * Compression codec for the ZSTD algorithm.
 */
public class ZstdCompressionCodec extends AbstractCompressionCodec {

 @Override
 protected ArrowBuf doCompress(BufferAllocator allocator, ArrowBuf uncompressedBuffer) {
 long maxSize = Zstd.compressBound(uncompressedBuffer.writerIndex());
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to cause too much waste of memroy in the worst case?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just a fact of life with compression algorithms. There are potentially ways of mitigating memory waste if changed the APIs around a little bit (trying to pack more data into a single ArrowBuf) or consolidating buffers into a larger buffer if compression level is high. But this is the guidance the documented methods give for sizing buffers.
 long dstSize = CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + maxSize;
 ArrowBuf compressedBuffer = allocator.buffer(dstSize);
 long bytesWritten = Zstd.compressUnsafe(
 compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH, dstSize,
 /*src*/uncompressedBuffer.memoryAddress(), /*srcSize=*/uncompressedBuffer.writerIndex(),
 /*level=*/3);
@liyafan82liyafan82Mar 29, 2021

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make the level configurable?
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at some point, the default is 3. LZ4 also has a compression level which I don't think we are setting.
@emkornfieldemkornfieldMar 29, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open up a JIRA to track configuration.
 if (Zstd.isError(bytesWritten)) {
 compressedBuffer.close();
 throw new RuntimeException("Error compressing: " + Zstd.getErrorName(bytesWritten));
 }
 compressedBuffer.writerIndex(CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH + bytesWritten);
 return compressedBuffer;
 }

 @Override
 protected ArrowBuf doDecompress(BufferAllocator allocator, ArrowBuf compressedBuffer) {
 long decompressedLength = readUncompressedLength(compressedBuffer);
 ArrowBuf uncompressedBuffer = allocator.buffer(decompressedLength);
 long decompressedSize = Zstd.decompressUnsafe(uncompressedBuffer.memoryAddress(), decompressedLength,
 /*src=*/compressedBuffer.memoryAddress() + CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH,
 compressedBuffer.writerIndex() - CompressionUtil.SIZE_OF_UNCOMPRESSED_LENGTH);
 if (Zstd.isError(decompressedSize)) {
 uncompressedBuffer.close();
 throw new RuntimeException("Error decompressing: " + Zstd.getErrorName(decompressedLength));
 }
 if (decompressedLength != decompressedSize) {
 uncompressedBuffer.close();
 throw new RuntimeException("Expected != actual decompressed length: " + 
 decompressedLength + " != " + decompressedSize);
 }
 uncompressedBuffer.writerIndex(decompressedLength);
 return uncompressedBuffer;
 }

 @Override
 public CompressionUtil.CodecType getCodecType() {
 return CompressionUtil.CodecType.ZSTD;
 }
}

4 changes: 4 additions & 0 deletions
java/compression/src/test/java/org/apache/arrow/compression/TestCompressionCodec.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -80,6 +80,10 @@ public static Collection<Object[]> getCodecs() {

 CompressionCodec lz4Codec = new Lz4CompressionCodec();
 params.add(new Object[]{lz4Codec.getCodecType(), len, lz4Codec});

 CompressionCodec zstdCodec = new ZstdCompressionCodec();
 params.add(new Object[]{zstdCodec.getCodecType(), len, zstdCodec});

 }
 return params;
 }
Expand Down