Getting Started with the Zstd and Lz4 Components

Requirements: IPWorks Zip

Introduction

IPWorks Zip includes two components for streaming compression and decompression: Zstd and Lz4. Unlike the archive-file components in IPWorks Zip (such as Zip or Gzip), these components have no concept of file names, archive headers, or multiple entries. Instead, they operate purely on a blob of data or a stream, making them well suited for compressing data in memory or over a network connection.

Zstd implements the Zstandard compression format (RFC 8878), and Lz4 implements the LZ4 Frame Format. Both are modern, general purpose compression formats designed for speed, and both components produce and consume standard, spec-compliant output. Data compressed by either component can be decompressed by any other compliant implementation of the format (such as the official zstd or lz4 command-line tools), and vice versa.

Zstd and Lz4 share the same API. Aside from the CompressionLevel range and the underlying format, everything in this article applies equally to both components.

Choosing Between Zstd and Lz4

Lz4 favors raw speed over compression ratio, making it a good choice when throughput matters most, such as compressing data on the fly over a fast network link or in a performance-sensitive pipeline.

Zstd generally achieves a better compression ratio than Lz4 for a similar amount of effort, at some cost to speed. It is a good default choice when reducing the size of the compressed data is more important than maximizing throughput.

If you are unsure which to use, Zstd is a reasonable default, and it is always possible to switch to Lz4 if compression speed becomes a bottleneck.

Getting Started

Both components expose the following core API members:

• CompressionLevel - the compression level to use.
• InputData / InputDataB - the data to compress or decompress (string or raw bytes).
• OutputData / OutputDataB - the resulting data, once available.
• CompressData / DecompressData - operate on a complete blob of data.
• CompressBlock / DecompressBlock - operate on data as they become available, in blocks (chunks).
• GetCompressionStream / GetDecompressionStream - return a stream that compresses or decompresses data as they are written to or read from an underlying stream (in applicable editions).
• CompressedData / DecompressedData - events that fire with data as they are produced.
• Reset - resets the component to its default state.

The sections below cover several ways of working with the data in more detail.

Compressing and Decompressing Complete Data

When the entire blob of data to compress or decompress is available ahead of time, the CompressData and DecompressData methods are the simplest way to use either component. Set the InputData property (or InputDataB for raw bytes) to the data, call the appropriate method, and the result will be available in the OutputData or OutputDataB properties.

The following example compresses and then decompresses a byte array using either component:

component.InputDataB = originalData; component.CompressData(); byte[] compressedData = component.OutputDataB;
component.Reset(); component.InputDataB = compressedData; component.DecompressData(); byte[] decompressedData = component.OutputDataB;

In addition to the OutputData property, the resulting data are also made available via the CompressedData and DecompressedData events as they become available.

Compressing and Decompressing in Blocks

When data become available incrementally (e.g., as they are read from a network connection), the CompressBlock and DecompressBlock methods may be used to process them in blocks (chunks) as each one arrives.

Note that the OutputData property is not applicable when using these methods; instead, the resulting data are made available exclusively via the CompressedData and DecompressedData events.

CompressBlock

Set the InputData property (or InputDataB) to the current block to be compressed and then call the CompressBlock method. This method takes a LastBlock boolean parameter, which should be enabled for the final block and disabled for every other block.

component.OnCompressedData += (s, e) => { // Handle e.DataB, which is the current chunk of compressed data. };
component.InputDataB = firstChunk; component.CompressBlock(false); // Not the last block.
component.InputDataB = secondChunk; component.CompressBlock(true); // The last block.

Note that the CompressedData event may not fire for every call to the CompressBlock method, since the underlying compression algorithm may need to accumulate more data before it can produce output.

DecompressBlock

Set the InputData property (or InputDataB) to the current block of compressed data and then call the DecompressBlock method. Unlike the CompressBlock method, no LastBlock parameter is required.

component.OnDecompressedData += (s, e) => { // Handle e.DataB, which is the current chunk of decompressed data. };
component.InputDataB = firstCompressedChunk; component.DecompressBlock();
component.InputDataB = secondCompressedChunk; component.DecompressBlock();

As with the CompressBlock method, the DecompressedData event may not fire for every call.

Compressing and Decompressing Streams

In the .NET, Java, and JavaScript editions, the GetCompressionStream and GetDecompressionStream methods may be used to wrap an existing stream, returning a new stream that transparently compresses or decompresses data as they pass through.

The GetCompressionStream method takes the destination stream as a parameter and returns a stream that uncompressed data should be written to. The component then compresses the data and writes them to the destination stream.

FileStream destination = new FileStream("output.zst", FileMode.Create); Stream compressionStream = component.GetCompressionStream(destination); compressionStream.Write(dataToCompress, 0, dataToCompress.Length); compressionStream.Close();

The GetDecompressionStream method works the other way around and takes a stream containing compressed data as a parameter and returns a stream that compressed data can be read from, yielding decompressed data.

FileStream source = new FileStream("output.zst", FileMode.Open); Stream decompressionStream = component.GetDecompressionStream(source); int bytesRead = decompressionStream.Read(buffer, 0, buffer.Length); decompressionStream.Close();

By default, closing a stream returned by either method also closes the underlying stream that was passed in. This can be controlled with the CloseBaseStream property.

Choosing a Compression Level

The CompressionLevel property controls the trade-off between compression speed and compression ratio. Higher values compress better but more slowly, and lower values compress faster but less effectively.

When using the Zstd component, note that some levels may produce functionally identical results to a neighboring level. A higher level will never compress worse than a lower one, but it is not guaranteed to compress better at every step. In practice, this means that it usually isn't necessary to use the maximum level to get most of the available benefit.