Data Compression

The byte-compression API transforms byte values and borrowed byte streams with LZ4, Deflate, bzip2, LZMA, or Zstandard. Algorithm, representation, effort, and decompression policy are fixed when the codec object is constructed. See Compressing Data for format selection, codec characteristics, and examples.

Configured Compression

ByteCompressor accepts an algorithm, a CompressionFormat, and a portable CompressionLevel. compress() accepts either an independent ByteBlock or a borrowed input and output stream. Each call creates independent codec state. maximumCompressedLength() includes the selected format’s framing overhead.

ByteDecompressor stores the matching algorithm and format plus DecompressionOptions. The options provide optional exact output-length validation and finite maximum output and workspace lengths. decompress() accepts a byte value or borrowed input and output streams. The configured objects can be reused without resetting them.

Representations

Raw produces the codec’s standard standalone payload. Zip produces the bytes for one ZIP entry’s compression method, not a local header, central directory, or complete archive. Core frames one continuous Raw payload in a version-2 ELBC envelope. Lengths and CRC-32 follow the payload, so neither input length nor output length must be known in advance. Core decoding requires the caller’s configured algorithm to match the embedded identifier.

Errors and Resource Policies

Invalid constructor combinations raise ParameterFailure. Malformed payloads, algorithm mismatches, unsupported stream features, and exact-length mismatches raise CompressionError. A payload that exceeds the configured output or workspace policy raises OutOfRangeError before the disallowed reservation.

Streaming Contract

CompressionTransferOptions selects an optional exact input prefix, positive buffer length (64 KiB by default), progress observer, and whole-value fallback policy. A finite prefix is consumed exactly; premature EOF is an error. Without a prefix, input is consumed through EOF. Decoders reject trailing or concatenated representations. The returned CompressionTransferResult contains cumulative input and output lengths. Borrowed streams remain open and are not flushed. Accepted output can still be queued; the caller must flush or close its destination to establish delivery.

Deflate, bzip2, LZMA, and Zstandard stream natively in their supported representations. Lz4Block uses capped whole-value buffering. supportsStreaming() reports this distinction separately on each configured object. RequireStreaming rejects a fallback configuration before reading input. Fallback input and output each have a default 256 MiB cap and may coexist with codec workspace. One-shot methods explicitly collect a complete result and do not use this transfer fallback cap.

Progress observers receive cumulative bytes read and accepted, optional totals, and a processing, finalizing, or completed phase. Intermediate callbacks may return Cancel. Callback exceptions propagate. Completion means codec validation and output acceptance succeeded, not that the destination was flushed. Timeouts use the input/output stream settings fixed at stream creation. A timeout terminates the transfer. Output already accepted before an error, timeout, or cancellation remains provisional and cannot be rolled back.

CompressionOptions sets compressor workspace (256 MiB by default). Decoder workspace defaults to 64 MiB. Total decoded output limits remain independent of working memory and must be raised explicitly for larger files. Codec windows, block storage, indexes, and scratch storage are bounded independently of total input length. Caller-owned storage and stream queues have separate limits. Internal codecs use synchronous pull readers and bounded output callbacks. They preserve decoding state across I/O chunks; they do not expose resumable sessions. Compression and decompression APIs are unsuitable for sensitive data. They ignore input ByteBlock sensitivity flags and return ordinary, unmarked byte blocks. Codec buffers, dictionaries, probability models, and scratch allocations are not securely erased, including on failure. The caller’s input retains its original sensitivity flag.

Core Wire Format

All integer fields use little-endian order. Version 1 is rejected.

• Header: four bytes ELBC, version byte 2, algorithm byte, and two zero reserved bytes.

• Payload: repeated 32-bit chunk lengths followed by that many payload bytes. Valid lengths are 1 through 65,536.

• Terminator: a zero 32-bit chunk length.

• Trailer: 64-bit uncompressed length, 64-bit payload length excluding framing, and 32-bit CRC-32 of uncompressed bytes.

Writers fill 64 KiB chunks except the final chunk. Chunk boundaries do not reset codec history. Decoders validate framing, codec completion, cumulative lengths, CRC-32, and resource limits.

Interface

class ByteCompressor

Compress complete byte values using one fixed algorithm, representation, and effort level.

Sensitive input flags are ignored; codec storage is not securely erased. Unsuitable for sensitive data.

See: Data Compression

Public Functions

ByteCompressor(CompressionAlgorithm algorithm, CompressionFormat format, CompressionLevel level = CompressionLevel::Default, CompressionOptions options = {})

Create a configured byte compressor.

Parameters:

• algorithm – The compression algorithm. Must support the selected format.

• format – The representation format. Must be supported by the selected algorithm.

• level – The compression effort level. Must be valid.

• options – The codec configuration options.

mem::ByteBlock compress(const mem::ByteBlock &data) const

Compress a byte block using the configured representation.

Throws:

err::OutOfRangeError – If processing exceeds a configured workspace, buffered-input, or output limit.

unit::ByteCount maximumCompressedLength(unit::ByteCount length) const

Calculate the upper bound for a compressed result in the configured representation.

Parameters:

length – The uncompressed length. Must be finite and yield a representable compressed-length bound.

auto compress(stream::ByteInputStream &source, stream::ByteOutputStream &destination, CompressionTransferOptions options = {}) const -> CompressionTransferResult

Transform one source prefix into a borrowed destination; neither stream is closed or flushed.

Throws:

• CompressionError – On invalid data, cancellation, or timeout. Output may be partial.

• err::OutOfRangeError – If processing exceeds a configured workspace, buffered-input, or output limit.

bool supportsStreaming() const noexcept

Test whether this configuration uses bounded native streaming.

CompressionAlgorithm algorithm() const noexcept

Get the selected compression algorithm.

CompressionFormat format() const noexcept

Get the selected compressed representation.

CompressionLevel level() const noexcept

Get the selected compression effort level.

class ByteDecompressor

Decompress complete byte values using one fixed algorithm, representation, and set of limits.

Sensitive input flags are ignored; codec storage is not securely erased. Unsuitable for sensitive data.

See: Data Compression

Public Functions

ByteDecompressor(CompressionAlgorithm algorithm, CompressionFormat format, DecompressionOptions options = {})

Create a configured byte decompressor.

Parameters:

• algorithm – The compression algorithm. Must support the selected format.

• format – The representation format. Must be supported by the selected algorithm.

• options – The decompression limits. Output and workspace limits must be finite; any expected output length must be finite and within the output limit. Raw LZ4 requires an expected output length.

mem::ByteBlock decompress(const mem::ByteBlock &data) const

Decompress a byte block using the configured representation and limits.

Parameters:

data – The compressed bytes. Decompression must fit within the configured output and workspace limits.

Throws:

• CompressionError – If the compressed representation is invalid.

• err::OutOfRangeError – If processing exceeds a configured workspace, buffered-input, or output limit.

auto decompress(stream::ByteInputStream &source, stream::ByteOutputStream &destination, CompressionTransferOptions options = {}) const -> CompressionTransferResult

Transform one source prefix into a borrowed destination; neither stream is closed or flushed.

Throws:

• CompressionError – On invalid data, cancellation, or timeout. Output may be partial.

• err::OutOfRangeError – If processing exceeds a configured workspace, buffered-input, or output limit.

bool supportsStreaming() const noexcept

Test whether this configuration uses bounded native streaming.

CompressionAlgorithm algorithm() const noexcept

Get the selected compression algorithm.

CompressionFormat format() const noexcept

Get the selected compressed representation.

inline const DecompressionOptions &options() const noexcept

Get the decompression limits and validation options.

class CompressionAlgorithm

A byte-compression algorithm supported by the library.

Raw algorithm values are stable because they are stored in compression envelopes.

See: Data Compression

Public Types

enum Value

The raw compression algorithm value.

Values:

enumerator Lz4Block

Standard raw LZ4 block compression.

enumerator Deflate

RFC 1951 Deflate compression.

enumerator Bzip2

Bzip2 block-sorting compression.

enumerator Lzma

LZMA1 compression.

enumerator Zstandard

Zstandard compression.

Public Functions

constexpr CompressionAlgorithm() noexcept = default

Create the default LZ4 block algorithm.

inline constexpr CompressionAlgorithm(const Value value) noexcept

Create an algorithm from its raw value.

inline constexpr Value toRawValue() const noexcept

Get the stable raw algorithm value.

text::String toString() const

Convert the algorithm to its stable lowercase identifier.

Public Static Functions

static std::optional<CompressionAlgorithm> fromString(const text::String &text) noexcept

Parse an exact lowercase algorithm identifier.

static CompressionAlgorithm fromStringOrThrow(const text::String &text)

Parse an exact lowercase algorithm identifier.

Throws:

err::ParseError – If the identifier is unsupported.

static std::span<const CompressionAlgorithm> all() noexcept

Get all supported algorithms in stable order.

class CompressionError : public erbsland::err::RuntimeError

A malformed or unsupported byte-compression representation.

Public Functions

CompressionError(CompressionErrorReason reason, text::String message) noexcept

Create a byte-compression error.

CompressionError(CompressionErrorReason reason, std::string_view message) noexcept

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

explicit CompressionError(CompressionErrorContext context) noexcept

Create a byte-compression error from complete diagnostic context.

virtual err::DiagnosticConstPtr diagnostic() const override

Create the complete structured diagnostic.

inline const CompressionErrorContext &context() const noexcept

Get the complete error context.

inline CompressionErrorReason reasonCode() const noexcept

Get the machine-readable failure reason.

inline const text::String &title() const noexcept

Get the concise error title.

inline const text::String &description() const noexcept

Get the detailed error description.

class CompressionErrorContext

User-facing context for a byte-compression error.

Public Functions

inline explicit CompressionErrorContext(CompressionErrorReason reason, text::String title, text::String description = {}) noexcept

Create context for a byte-compression error.

Parameters:

• reason – The machine-readable failure reason.

• title – A concise description of what went wrong.

• description – An optional explanation of why the error occurred.

inline CompressionErrorReason reason() const noexcept

Get the machine-readable failure reason.

inline CompressionErrorContext &setReason(const CompressionErrorReason reason) noexcept

Set the machine-readable failure reason.

inline const text::String &title() const noexcept

Get the concise error title.

inline CompressionErrorContext &setTitle(text::String title) noexcept

Set the concise error title.

inline const text::String &description() const noexcept

Get the detailed error description.

inline CompressionErrorContext &setDescription(text::String description) noexcept

Set the detailed error description.

enum class erbsland::compression::CompressionErrorReason : uint8_t

A machine-readable byte-compression failure reason.

Values:

enumerator Cancelled

A progress callback cancelled the transfer.

enumerator Timeout

A stream operation timed out.

enumerator MalformedData

The compressed payload is malformed or truncated.

enumerator LengthMismatch

Encoded and expected lengths do not match.

enumerator UnsupportedAlgorithm

The selected or encoded algorithm is unsupported.

enumerator AlgorithmMismatch

The encoded algorithm differs from the configured algorithm.

enumerator UnsupportedFeature

The payload uses a recognized but unsupported codec feature.

enumerator UnsupportedEnvelopeVersion

The envelope version is unsupported.

enum class erbsland::compression::CompressionFallbackPolicy

Policy for algorithms requiring complete byte values.

Values:

enumerator AllowBuffered

Allow one-shot codecs within the fallback buffer limits.

enumerator RequireStreaming

Reject unsupported streaming before consuming input.

enum class erbsland::compression::CompressionFormat : uint8_t

The outer representation of compressed bytes.

Values:

enumerator Raw

The algorithm’s standalone representation.

enumerator Zip

The payload representation used in a ZIP entry.

enumerator Core

The self-describing Erbsland Core compression envelope.

enum class erbsland::compression::CompressionLevel : uint8_t

A portable, human-facing compression effort level.

Values:

enumerator Fastest

Minimize compression time.

enumerator Fast

Prefer compression speed over size.

enumerator Default

Use the balanced codec-specific default.

enumerator High

Prefer compressed size over speed.

enumerator Highest

Use the highest supported compression effort.

class CompressionOptions

Compressor memory policy.

Public Functions

inline unit::ByteCount maximumWorkspaceLength() const noexcept

Get the maximum codec workspace.

inline CompressionOptions &setMaximumWorkspaceLength(unit::ByteCount value)

Set the finite maximum codec workspace.

enum class erbsland::compression::CompressionPhase

Current transfer phase.

Values:

enumerator Processing

Input and output are being transferred.

enumerator Finalizing

The representation is being completed and validated.

enumerator Completed

All output was accepted and validation succeeded.

struct CompressionProgress

Progress of a synchronous compression transfer.

Public Members

CompressionTransferResult transferred

Cumulative I/O counts.

std::optional<unit::ByteCount> inputTotal

Exact input length, if supplied.

std::optional<unit::ByteCount> outputTotal

Expected output length, if known.

CompressionPhase phase = {CompressionPhase::Processing}

Current phase.

using erbsland::compression::CompressionProgressFn = std::function<CompressionProgressAction(const CompressionProgress&)>

Synchronous observer; cancellation is honored before completion.

enum class erbsland::compression::CompressionProgressAction

Action requested by a progress observer.

Values:

enumerator Continue

Continue the operation.

enumerator Cancel

Terminate an unfinished operation with a cancellation error.

class CompressionTransferOptions

I/O and buffering policy for one borrowed-stream transfer.

Public Functions

inline const std::optional<unit::ByteCount> &inputLength() const noexcept

Get the optional exact input prefix length.

inline CompressionTransferOptions &setInputLength(unit::ByteCount value)

Set the exact input prefix length.

inline CompressionTransferOptions &clearInputLength()

Clear the input prefix constraint.

inline unit::ByteCount bufferLength() const noexcept

Get the transfer buffer size.

inline CompressionTransferOptions &setBufferLength(unit::ByteCount value)

Set the positive finite transfer buffer size.

inline unit::ByteCount maximumBufferedLength() const noexcept

Get the limit for each complete fallback value.

inline CompressionTransferOptions &setMaximumBufferedLength(unit::ByteCount value)

Set the fallback input and output limit.

inline CompressionFallbackPolicy fallbackPolicy() const noexcept

Get the fallback policy.

inline CompressionTransferOptions &setFallbackPolicy(CompressionFallbackPolicy value)

Set the fallback policy.

inline const CompressionProgressFn &progress() const noexcept

Get the progress observer.

inline CompressionTransferOptions &setProgress(CompressionProgressFn value)

Set the progress observer.

struct CompressionTransferResult

Final byte counts accepted by a compression transfer.

Public Members

unit::ByteCount inputLength

Bytes read from the source.

unit::ByteCount outputLength

Bytes accepted by the destination.

class DecompressionOptions

Limits and validation options for one byte decompressor.

Workspace covers dynamically allocated codec dictionaries, tables, indexes, and scratch buffers. It excludes the compressed input, returned output, stack storage, and allocator metadata.

See: Data Compression

Public Functions

inline constexpr const std::optional<unit::ByteCount> &expectedOutputLength() const noexcept

Get the optional exact decompressed output length.

inline constexpr DecompressionOptions &setExpectedOutputLength(unit::ByteCount value) noexcept

Set the exact decompressed output length to validate.

inline constexpr DecompressionOptions &clearExpectedOutputLength() noexcept

Clear exact output-length validation.

inline constexpr unit::ByteCount maximumOutputLength() const noexcept

Get the maximum decompressed output length.

inline constexpr DecompressionOptions &setMaximumOutputLength(unit::ByteCount value) noexcept

Set the maximum decompressed output length.

inline constexpr unit::ByteCount maximumWorkspaceLength() const noexcept

Get the maximum codec workspace length.

inline constexpr DecompressionOptions &setMaximumWorkspaceLength(unit::ByteCount value) noexcept

Set the maximum codec workspace length.

Public Static Attributes

static constexpr auto cDefaultMaximumOutputLength = unit::ByteCount{256U * 1024U * 1024U}

The default maximum decompressed output length.

static constexpr auto cDefaultMaximumWorkspaceLength = unit::ByteCount{64U * 1024U * 1024U}

The default maximum codec workspace length.