Add locks for de-/compressor + initialize decompressor

[svilupp]

Fixes#396

This PR proposes to:

• fix the issue with indexing by Threads.threadid() by locking the compressor/decompressor (see Error/Segfault when writing many partitions #396)
• initialize an array of decompressors, similar to what has been previously done for compressors (eg,

  arrow-julia/src/Arrow.jl

  Line 80 in 9b36c8b

  CodecZstd.TranscodingStreams.initialize(zstd)

  )

Acquiring locks introduces some additional overhead (I was not confident we could use SpinLocks given we traverse several packages and into C-code, so I opted for a slightly slower ReentrantLock).

I've created a minimalistic workload (100 partitions with 1 row) to see mostly the lock overhead.
I couldn't detect any slowdown in writing speed.
Since we're initializing decompressors upfront, we get c. 30% speed up when reading (no need to initialize for each record batch)

using Arrow, Tables, Random
# Speed test N = 100
len = 100
t = Tables.rowtable((; x1 = map(x -> randstring(len), 1:N)));
fn="test_partitioned.arrow"
## Before
# Write:
@btime Arrow.write($fn, Iterators.partition($t, 1); compress = :lz4)
# 2.067 ms (12912 allocations: 13.24 MiB)
# Read:
@btime t=Arrow.Table($fn);
# 718.458 μs (9305 allocations: 533.62 KiB)
## After
# Write:
@btime Arrow.write($fn, Iterators.partition($t, 1); compress = :lz4)
# 2.021 ms (12987 allocations: 13.24 MiB)
# Read:
@btime t=Arrow.Table($fn);
# 461.083 μs (9712 allocations: 543.52 KiB)

[baumgold]

This function should probably be broken down into a few smaller functions for clarity:

toarrowvector(a::ArrowVector, compression::Nothing, kw...) = a
functiontoarrowvector(a::ArrowVector, compression::AbstractVector{LZ4FrameCompressor}, kw...)
lock(LZ4_FRAME_COMPRESSOR_LOCK[Threads.threadid()]) docompress(Meta.CompressionTypes.LZ4_FRAME, compression[tid], a)
endendfunctiontoarrowvector(a::ArrowVector, compression::LZ4FrameCompressor, Vector{ZstdCompressor}, ZstdCompressor}=nothing, kw...)
compress(Meta.CompressionTypes.LZ4_FRAME, compression, a)
endfunctiontoarrowvector(a::ArrowVector, compression::AbstractVector{ZstdCompressor}, kw...)
compress(Meta.CompressionTypes.ZSTD, compression, a)
endfunctiontoarrowvector(a::ArrowVector, compression::ZstdCompressor, kw...)
lock(ZSTD_COMPRESSOR_LOCK[Threads.threadid()]) docompress(Meta.CompressionTypes.ZSTD, compression[tid], a)
endendfunctiontoarrowvector(x, i=1, de=Dict{Int64, Any}(), ded=DictEncoding[], meta=getmetadata(x); compression::Union{Nothing, Vector{LZ4FrameCompressor}, LZ4FrameCompressor, Vector{ZstdCompressor}, ZstdCompressor}=nothing, kw...)
@debugv2"converting top-level column to arrow format: col = $(typeof(x)), compression = $compression, kw = $(kw.data)"@debugv3 x
A =arrowvector(x, i, 0, 0, de, ded, meta; compression=compression, kw...)
A =toarrowvector(A, compression)
@debugv2"converted top-level column to arrow format: $(typeof(A))"@debugv3 A
return A
end

[baumgold]

A single thread can only do one thing at a time so maybe we can have a single lock per thread to be shared amongst all compressors/decompressors?

[baumgold]

Maybe we should be a bit lazier about initializing these as I believe they're quite expensive. Shall we initialize them only as needed? For example, if a user is only decompressing (reading) data then there's no reason to pay the penalty of initializing compressors (and vice versa). Same idea for LZ4 vs ZSTD - if a user only uses one then no need to initialize both.

[quinnj]

Not sure if @svilupp wasn't able to see this through, but I've cleaned up their code here: #445. I can confirm that I was able to reproduce the original segfault on Apple M1 and that the code in that PR resolves the issue. I believe that PR should also address @baumgold's comments from this PR.