LZ4 Blocks

LZ4 is designed for the moments when decompression should feel almost invisible. It replaces nearby repeated byte sequences with short backward references and otherwise leaves bytes as literals. There is no statistical model to rebuild and no entropy-coded bit stream to navigate, so a decoder can spend most of its time copying bytes.

This page builds a raw LZ4 block from a small repeated pattern, follows its fields on the wire, and explains why the algorithm is such a good fit for independent application records. It also shows where LZ4 gives up compression ratio in exchange for speed and how Erbsland Core’s five portable levels change the match search.

Understanding LZ4

From Literals to Matches

Imagine that an instrument produces the bytes ABCDABCDABCD. The first ABCD has not appeared before, so the compressor writes those four bytes as literals. When the next ABCD begins, the same sequence is only four bytes behind the current position. Instead of writing eight more literal bytes, the compressor can describe one match with distance four and length eight:

input                        A B C D         A B C D A B C D
                            └── literals ──┘└──── match ────┘

decoded output               A B C D       A B C D  A B C D
                            └── source ──┘└─ copied twice ─┘
                                  ▲               │
                                  └─── offset 4 ──┘

The source and destination are allowed to overlap. After the decoder copies the first four match bytes, those newly produced bytes become part of the source for the next four. This small rule also makes long runs inexpensive: a match with offset one can repeat the preceding byte many times.

Every match must be at least four bytes long. Shorter repetitions would rarely save space after their length and offset were recorded. The offset is an unsigned 16-bit value, so a match can refer to bytes between one and 65,535 positions behind the current output. Useful repetition beyond that window must be written literally or found again closer to the current position.

Sequences and Tokens

A raw block is a series of sequences. Each regular sequence contains some literals followed by one match; the final sequence contains literals only. The one-byte token keeps common lengths compact by dividing itself into two four-bit fields:

bit              7             4 3             0
                +----------------+---------------+
token           | literal length | match length  |
                +----------------+---------------+
                   high nibble      low nibble

sequence        +-------+-------------+----------+--------+--------------+
                | token | lit. length | literals | offset | match length |
                +-------+-------------+----------+--------+--------------+
                            optional              optional
byte order                                                 little endian

The upper nibble starts the literal length. Values below 15 are complete; a value of 15 means extension bytes follow before the literals. Each extension byte contributes its value, and a byte containing 255 means another extension byte follows. The lower nibble describes the match length in the same way, except that four is added because every match already has the minimum length of four.

After the literals, the two-byte little-endian offset points backward from the current output position. An offset of zero is invalid. The final sequence stops after its literals, so it has neither an offset nor a match-length extension. The end of the compressed block must come from the surrounding container; the raw format has no end marker, output size, checksum, or frame header.

Representations and Memory

CompressionFormat::Raw produces exactly this standard block representation. Because the decoder cannot learn the original size from the block, a raw LZ4 ByteDecompressor requires DecompressionOptions::expectedOutputLength. That value normally comes from a trusted record header or schema and is checked exactly.

CompressionFormat::Core places the block inside the ELBC envelope described on the main compression page. The envelope supplies the algorithm identifier, original length, and compressed boundary that the raw block omits. CompressionFormat::Zip is not available for LZ4 because this API does not define a ZIP method for raw LZ4 blocks.

Decoding needs no separate dictionary allocation: previously produced output is the match history. This keeps codec workspace small, although the compressed input and returned output naturally still occupy memory. The decoder validates every offset and length before copying, rejects trailing data, and requires the last sequence to end exactly at the supplied block boundary.

The raw layout is defined by the LZ4 Block Format Description.

Pros and Cons

Where LZ4 Performs Well

LZ4 is a natural companion for caches, network messages, generated resources, and serialized records that are decoded often or on a latency-sensitive path. Logs and structured binary data tend to repeat field names, tags, padding, and common values within the 64 KiB match window. The decoder turns those matches back into bytes with straightforward copies, which is especially attractive when one piece of data is written once but read many times.

Independent blocks also give an application useful recovery boundaries. A damaged record does not require rebuilding a large statistical state from everything that came before it. With Core framing, the application receives this speed while also getting an exact size and a clear algorithm identity.

Where Another Algorithm May Fit Better

The same simplicity limits LZ4’s compression ratio. It does not assign shorter codes to common literal values, and it cannot reference repetitions more than 65,535 bytes away. Deflate adds Huffman coding, while bzip2, LZMA, and Zstandard use richer transformations or models that can describe some data more compactly.

Tiny values can grow because even a token and length information have a cost. Random, encrypted, and already compressed bytes usually have no useful matches at all. For long-term archives, where encoding happens rarely and storage cost dominates, LZMA or another deeper compressor may justify its additional work. Raw LZ4 is also a poor fit when the surrounding format cannot provide an exact original length.

Compression Levels

CompressionLevel controls how much work the compressor spends looking for a useful match at each position. The resulting bytes do not record this abstract level, and the decoder does not need to know it. All levels produce the same standard raw block representation.

LZ4 compression-level mapping

Core level	Match-chain depth	Search behaviour
Fastest	1 candidate	Accept the first useful nearby match and minimize search work.
Fast	Up to 4 candidates	Spend a little more time comparing recent alternatives.
Default	Up to 16 candidates	Balance match quality and compression time for general records.
High	Up to 64 candidates	Search more deeply and look ahead one byte before committing to a match.
Highest	Up to 256 candidates	Use the deepest search and the same lazy-match decision for compact output.

A deeper search can find a longer or closer repetition, but it is not a promise that every higher level produces fewer bytes. The best result still depends on the input, and a small value may offer every level exactly the same useful match. The following compiled demo compresses one observation at all five levels and verifies every result with a decoder that knows only the algorithm, representation, and expected output length.

compression/ByteCompression/CompareCompressionLevels.cpp

void compareCompressionLevels(
    const el::CompressionAlgorithm algorithm,
    const el::CompressionFormat format,
    const el::StringLiteral algorithmName) {
    const auto readings = createObservatoryReadings();
    const auto options = el::DecompressionOptions{}
                             .setExpectedOutputLength(readings.length())
                             .setMaximumOutputLength(el::ByteCount{16U * 1024U})
                             .setMaximumWorkspaceLength(el::ByteCount{64U * 1024U * 1024U});

    el::io::printLine("Instrument        : radiotelescópio"_el);
    el::io::printLine("Algorithm         : "_el, algorithmName);
    el::io::printLine("Original bytes    : "_el, readings.length().toSizeT());

    // Compress and restore the same observation at every portable effort level.
    for (const auto &[level, name] : cCompressionLevels) {
        const auto compressor = el::ByteCompressor{algorithm, format, level};
        const auto compressed = compressor.compress(readings);
        const auto decompressor = el::ByteDecompressor{algorithm, format, options};
        const auto restored = decompressor.decompress(compressed);
        el::io::printLine(
            name,
            " bytes: "_el,
            compressed.length().toSizeT(),
            ", restored: "_el,
            el::BooleanFormat::yesNo(),
            restored == readings);
    }
}

void compareLz4Levels() {
    compareCompressionLevels(el::CompressionAlgorithm::Lz4Block, el::CompressionFormat::Raw, "lz4-block"_el);
}

Output

Instrument        : radiotelescópio
Algorithm         : lz4-block
Original bytes    : 3584
Fastest bytes: 58, restored: yes
Fast bytes: 41, restored: yes
Default bytes: 41, restored: yes
High bytes: 41, restored: yes
Highest bytes: 41, restored: yes

Complete demo sources

compression/ByteCompression/CompareCompressionLevels.cpp

// Copyright (c) 2026 Tobias Erbsland - https://erbsland.dev
// SPDX-License-Identifier: Apache-2.0

#include <DemoCommon.hpp>
#include <erbsland/BooleanFormat.hpp>
#include <erbsland/Byte.hpp>
#include <erbsland/ByteBlock.hpp>
#include <erbsland/ByteBlockEditor.hpp>
#include <erbsland/ByteCompressor.hpp>
#include <erbsland/ByteCount.hpp>
#include <erbsland/ByteDecompressor.hpp>
#include <erbsland/CompressionAlgorithm.hpp>
#include <erbsland/CompressionFormat.hpp>
#include <erbsland/CompressionLevel.hpp>
#include <erbsland/DecompressionOptions.hpp>
#include <erbsland/StandardStreams.hpp>
#include <erbsland/StringLiteral.hpp>

#include <array>

namespace demo {

struct CompressionLevelEntry final {
    el::CompressionLevel level;
    el::StringLiteral name;
};

constexpr auto cCompressionLevels = std::array<CompressionLevelEntry, 5U>{
    CompressionLevelEntry{el::CompressionLevel::Fastest, "Fastest"_el},
    CompressionLevelEntry{el::CompressionLevel::Fast, "Fast"_el},
    CompressionLevelEntry{el::CompressionLevel::Default, "Default"_el},
    CompressionLevelEntry{el::CompressionLevel::High, "High"_el},
    CompressionLevelEntry{el::CompressionLevel::Highest, "Highest"_el},
};

auto createObservatoryReadings() -> el::ByteBlock {
    auto readings = el::ByteBlockEditor{};
    for (auto cycle = 0U; cycle < 8U; ++cycle) {
        readings.append(el::Byte{0x18U}, el::ByteCount{192U});
        readings.append(el::Byte{0x2aU}, el::ByteCount{32U});
        readings.append(el::Byte{0x19U}, el::ByteCount{192U});
        readings.append(el::Byte{0x2aU}, el::ByteCount{32U});
    }
    return el::ByteBlock{readings};
}

/// Compare all portable compression levels for one explicitly selected algorithm.
///
/// A level changes only the compressor's effort. The decoder receives the
/// algorithm and representation, but it does not need the compression level.
void compareCompressionLevels(
    const el::CompressionAlgorithm algorithm,
    const el::CompressionFormat format,
    const el::StringLiteral algorithmName) {
    const auto readings = createObservatoryReadings();
    const auto options = el::DecompressionOptions{}
                             .setExpectedOutputLength(readings.length())
                             .setMaximumOutputLength(el::ByteCount{16U * 1024U})
                             .setMaximumWorkspaceLength(el::ByteCount{64U * 1024U * 1024U});

    el::io::printLine("Instrument        : radiotelescópio"_el);
    el::io::printLine("Algorithm         : "_el, algorithmName);
    el::io::printLine("Original bytes    : "_el, readings.length().toSizeT());

    // Compress and restore the same observation at every portable effort level.
    for (const auto &[level, name] : cCompressionLevels) {
        const auto compressor = el::ByteCompressor{algorithm, format, level};
        const auto compressed = compressor.compress(readings);
        const auto decompressor = el::ByteDecompressor{algorithm, format, options};
        const auto restored = decompressor.decompress(compressed);
        el::io::printLine(
            name,
            " bytes: "_el,
            compressed.length().toSizeT(),
            ", restored: "_el,
            el::BooleanFormat::yesNo(),
            restored == readings);
    }
}

/// Compare the five LZ4 compression levels using raw blocks.
void compareLz4Levels() {
    compareCompressionLevels(el::CompressionAlgorithm::Lz4Block, el::CompressionFormat::Raw, "lz4-block"_el);
}

/// Compare the five Deflate compression levels using RFC 1951 streams.
void compareDeflateLevels() {
    compareCompressionLevels(el::CompressionAlgorithm::Deflate, el::CompressionFormat::Raw, "deflate"_el);
}

/// Compare the five bzip2 compression levels using complete streams.
void compareBzip2Levels() {
    compareCompressionLevels(el::CompressionAlgorithm::Bzip2, el::CompressionFormat::Raw, "bzip2"_el);
}

/// Compare the five LZMA compression levels using LZMA-Alone streams.
void compareLzmaLevels() {
    compareCompressionLevels(el::CompressionAlgorithm::Lzma, el::CompressionFormat::Raw, "lzma"_el);
}

/// Compare the five Zstandard compression levels using standard frames.
void compareZstandardLevels() {
    compareCompressionLevels(el::CompressionAlgorithm::Zstandard, el::CompressionFormat::Raw, "zstandard"_el);
}

}