feat: Add support for LZ4 compression

[andygrove]

Which issue does this PR close?

Closes #1178

Builds on #1192 so we need to merge that PR first

Rationale for this change

LZ4 may provide faster compression than ZSTD, which could help with shuffle performance.

What changes are included in this PR?

How are these changes tested?

[Dandandan]

I am not really familiar with the code, but shouldn't StreamWriter and encoder be created only once per stream instead of per batch?

[andygrove]

Yes, that is something I have been thinking about as well. We have the cost of writing the schema for each batch currently, and the schema is guaranteed to be the same for each batch.

[andygrove]

To add some more context here, we buffer rows per partition until we reach the desired batch size and then need to serialize that batch to bytes that can be read as one block by CometBlockStoreShuffleReader.

[Dandandan]

Besides writing the schema each time, I guess it will also have higher overhead of flushing each time (less efficient buffering), having lower compressibility, etc. ?

[andygrove]

I filed #1186 for re-using the writer across many batches. It looks like a big perf win.

[andygrove]

Here are my findings from hacking on this today.

LZ4 provides two compression formats: LZ4 Block Format and LZ4 Frame Format.

Spark uses the Java library https://github.com/lz4/lz4-java and specifically uses LZ4BlockOutputStream which seems to be a proprietary streaming LZ4 format, as noted in the documentation:

/**
 * Streaming LZ4 (not compatible with the LZ4 Frame format).
 * This class compresses data into fixed-size blocks of compressed data.
 * This class uses its own format and is not compatible with the LZ4 Frame format.
 * For interoperability with other LZ4 tools, use {@link LZ4FrameOutputStream},
 * which is compatible with the LZ4 Frame format. This class remains for backward compatibility.
 * @see LZ4BlockInputStream
 * @see LZ4FrameOutputStream
 */
public class LZ4BlockOutputStream extends FilterOutputStream {

edit: Apache Commons provides FramedLZ4CompressorInputStream and BlockLZ4CompressorInputStream so I am testing with using those from CometBlockStoreShuffleReader instead of using Spark's codec.

[andygrove]

switched to lz4_flex crate:

shuffle_writer/shuffle_writer: encode and compress (zstd)
                        time:   [209.27 µs 211.21 µs 213.47 µs]
shuffle_writer/shuffle_writer: encode and compress (lz4 frame)
                        time:   [162.58 µs 165.17 µs 167.76 µs]
shuffle_writer/shuffle_writer: encode and compress (lz4 block)
                        time:   [180.16 µs 183.93 µs 188.09 µs]

[andygrove]

The lz4_flex crate should get even faster once PSeitz/lz4_flex#175 is merged

[andygrove]

Current status:

✔️ I can run real benchmarks
✔️ shuffle write encoding + compression time is improved
🤕 Overall query time is more than 10x slower, so perhaps there is an issue with the reader side now

[andygrove]

LZ4 support is now part of #1192