An URL Inputstream

[v-gu]

Hi Xerial,

Recently I'm investigating a JVM memory leaky problem that related to a Spark application, and one of the trace points to the snappy library call which is listed blow:

https://github.com/xerial/snappy-java/blob/develop/src/main/java/org/xerial/snappy/Snappy.java#L292

Should this inputstream be closed after use? if the resource is in Jar then the input stream would be GzipInputStream and it will allocate off-heap memory.

Don't really sure if this was rated to my problem, but I'm just curious here. Thank you.

[JoshRosen]

This looks like a legitimate leak to me. I think that Spark will hit this leak whenever it allocates a SnappyCompressionCodec (https://github.com/apache/spark/blob/29cecd4a42f6969613e5b2a40f2724f99e7eec01/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L154), which I think we may do on the order of once per task at least.

Therefore, I think we should fix this in both Snappy and Spark: Spark can mitigate the problem by only calling this method once and caching the result, while this library should fix the leak.

[v-gu]

CompressionCodec is used in a lot of places, checkpointing, shuffle, broadcasts, event logs, etc..

[xerial]

OK. I fixed this problem.
I will release it as 1.1.2.1

[xerial]

Deployed snappy-java-1.1.2.1 to the Maven central. Check it out in several minutes. Thanks.

[xerial]

Now snappy-java-1.1.2.1 is available. Thanks for reporting.