Display more specific error message when failing to decompress an empty stream

[JoshRosen]

In Apache Spark, I've run into problems where our own code mistakenly tries to decompress an empty stream using SnappyInputStream. In this case, snappy-java produces an error like

java.io.IOException: PARSING_ERROR(2)
 at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:84)
 at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
 at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:594)
 at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
 at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
 at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)

In SnappyInputStream.readHeader, there's a check that tests whether the header is shorter than expected or doesn't match the SnappyOutputStream magic header:

 // Quick test of the header 
 if (readBytes < header.length || header[0] != SnappyCodec.MAGIC_HEADER[0]) {
 // do the default uncompression
 readFully(header, readBytes);
 return;
 }

In the case where readBytes is 0, it would be helpful if snappy-java reported an error such as new IOException("Cannot decompress empty stream") instead of falling back on the regular Snappy decompression and throwing a confusing PARSING_ERROR.

[xerial]

Thanks for reporting problem. I haven't checked the case when the input is empty. I'll fix this problem.

[xerial]

I want to confirm whether throwing an IOException for an empty input is the expected behaviour? It is also possible to finish decompression without throwing exception if the input is empty.

[xerial]

@JoshRosen
I fixed the problem by not throwing IOException since it should be the default behaviour. You can check the emptiness by looking at the size of the written data (i.e. return values of int read(dst, offset, len))

Deployed this version as 1.1.1.4-SNAPSHOT:
https://oss.sonatype.org/content/repositories/snapshots/org/xerial/snappy/snappy-java/1.1.1.4-SNAPSHOT/

[JoshRosen]

According to the Snappy spec, is an empty stream / array a valid input to the decompressor? If I compress an empty byte array / stream, then shouldn't that produce non empty compressed output that consists only of the magic header? As far as I can tell, Snappy will never write an empty output and therefore attempting to decompress an empty stream is always an error.

On Oct 22, 2014, at 8:05 PM, "Taro L. Saito" notifications@github.com wrote:

I want to confirm whether throwing an IOException for an empty input is the expected behaviour? It is also possible to finish decompression without throwing exception if the input is empty.

—
Reply to this email directly or view it on GitHub.

[xerial]

Good point. Actually Snappy.compress(new byte[0]) produces 1-byte result. So SnappyInputStream must throw an exception. I'll modify the code. Thanks.

[xerial]

To capture the empty input stream error, I added SnappyIOException (extended from IOException), to make it easy to detect the error type by using the pattern matching in Scala.

Could you review the changes? Thanks in advance.

[JoshRosen]

This looks good to me; it will be a huge debugging aid for us, since it's kind of difficult / expensive to add these checks at higher levels of the stack. Thanks for your help!

[xerial]

Just released snappy-java-1.1.1.4. It will be available from Maven central in a few hours.

Thanks for the detailed report. It helps me a lot to understand the problem quickly.

[ash211]

Just want to thank you @xerial for the incredibly quick response on this issue! It's a big help for the Spark community that you're so on top of these sorts of things. I really appreciate it and I think many others do too.

You can see our struggles with identifying the cause of the Spark bug at SPARK-3630

[xerial]

@ash211
Thanks for letting me know how spark community handling this problem. It looks like they are doing very nice jobs for identifying the cause of the errors.

@JoshRosen
Are you using snappy-java-1.1.1.6, which fixes #92?

Please let me know if there is something I can do for resolving the issue. Releasing a new version of snappy-java can be done in my spare time.