Text Encoding and Conversion

String Converter

Introduction

StringConverter is the explicit conversion entry point for Erbsland Core strings, editors, literals and the supported standard-library string types. It keeps conversion helpers out of the core string classes while still allowing concise call sites.

auto text = el::StringConverter{std::string_view{"Hello"}}.toString();
auto utf16 = el::StringConverter{text}.toStdU16String();
auto converted = el::StringConverter{utf16}.toString();

toString() returns a read-only UTF-8 value. It aliases compatible Core storage in tolerant mode and owns newly converted storage. Every conversion method accepts EncodingMode, defaulting to Tolerant.

Extension libraries can support additional source types by specializing StringConverterTraits.

Encoding Mode

EncodingMode controls how encoding errors are handled by StringConverter. Tolerant is the default and replaces malformed input with Unicode replacement characters when transcoding. Compatible representations may instead be copied unchanged without validation. Strict throws EncodingError when malformed input is encountered.

Important

Strict conversion can be much slower than tolerant conversion, even for valid input. In particular, compatible representations that tolerant mode can share or copy directly must be validated in strict mode. Keep Tolerant for ordinary conversions and select Strict only when encoding validity must be enforced at this boundary, including when the source is already an Erbsland Core string. Examples include security-sensitive processing or untrusted network input that must be rejected if its encoding is invalid. At such a boundary, enforcing validity takes priority over the additional decoding time.

For practical conversion guidance, see Converting Strings.

String Decoder

StringDecoder decodes ByteBlock or ByteBlockEditor input into Erbsland Core strings.

auto text = el::StringDecoder{bytes}.decode(el::StringEncoding::Utf8);
auto utf16 = el::StringDecoder{bytes}.toU16String(el::StringEncoding::Utf16);
el::StringDecoder{bytes}.validateOrThrow(el::StringEncoding::Utf8);

The decoder accepts StringEncoding, StringBomMode, and EncodingMode for all target string widths. Only an encoded U+FEFF signature at the start of the byte input is interpreted as a BOM. For UTF-8 input with a UTF-8 result, tolerant decoding copies the payload bytes without validating or repairing them. Malformed sequences and repeated or embedded U+FEFF bytes remain in storage; character operations on the resulting string handle them tolerantly. Strict decoding validates the complete payload before copying it and rejects malformed sequences, including repeated or embedded U+FEFF. UTF-16/32 decoding continues to interpret and transcode the input, replacing malformed content in tolerant mode. The strict validation pass can make UTF-8 decoding much slower than the tolerant byte copy, even when all input is valid. This also applies to whole-file text reads through PathContent::readTextOrThrow(). Choose strict decoding only when this read is a required encoding-validation boundary. Incremental text-stream behavior and mode selection are described in Streams.

Use validateOrThrow() when encoded bytes must be checked without constructing a decoded string. It applies strict validation together with the selected encoding and BOM policy, traverses the input once, and allocates no output storage.

String Encoder

StringEncoder encodes Erbsland Core strings and editors into ByteBlock data or directly into a RingBuffer. Use encodedLength() to calculate the exact byte length without allocating an intermediate byte block. The encodeTo() method reserves the complete output and commits it atomically; insufficient capacity leaves readable ring data unchanged. The encoder is intentionally limited to Erbsland Core text sources.

auto bytes = el::StringEncoder{text}.encode(el::StringEncoding::Utf8, el::StringBomMode::Reject);
auto byteLength = el::StringEncoder{text}.encodedLength(el::StringEncoding::Utf16);
auto result = el::StringEncoder{text}.encodeTo(ring, el::StringEncoding::Utf16);

Each standalone encode() or encodeTo() call applies its requested BOM policy independently, including calls for empty text that produce only a BOM. Stateful output streams suppress the BOM after their first successful write. Explicit BOM output is generated as an encoding signature. When the source and target representations match, the encoder copies the native units directly without a validation pass because Erbsland Core strings are assumed to contain valid text. When transcoding is necessary, malformed source sequences become Unicode replacement characters. Call isEncodingValid() or encodingValidOrThrow() before encoding when invalid internal text must be detected. An ordinary Char{0xFEFF} in source content is never interpreted as a signature; during transcoding it becomes a replacement character like other invalid content.

StringEncoder accepts the read-only string and editor types for all three supported string widths.

String Kind

StringKind selects the concrete string encoding used by generic text construction APIs. Use it when user code should decide whether a result is built as UTF-8, UTF-16, or UTF-32 text.

Usage ^^^^^

Pass StringKind to AnyStringBuilder when you need an empty builder for a specific output encoding.

auto utf8 = el::AnyStringBuilder{el::StringKind::U8};
auto utf16 = el::AnyStringBuilder{el::StringKind::U16};
auto utf32 = el::AnyStringBuilder{el::StringKind::U32};

Any-Width String Values

AnyString stores a read-only UTF-8, UTF-16 or UTF-32 string without changing its width. It can be compared directly with another AnyString, a width-specific Core string or editor, or any _el string literal. Comparisons are lexicographical by decoded code point and do not create converted strings. Malformed encoded units are compared as Char::replacement().

auto pattern = el::AnyString{u"[a-z]+"_el};

if (pattern == "[a-z]+"_el) {
    // The UTF-16 pattern is compared directly with the UTF-8 literal.
}

An empty AnyString has no selected width. It compares equal to empty strings of every width and sorts before any non-empty string.

Standalone Conversion Helpers

The toString() overloads convert common text, numeric, boolean, and ordering values into the common String type. Boolean conversion accepts BooleanFormat. The std::strong_ordering overload returns less, equal, or greater.

String Decode Buffer

Introduction

StringDecodeBuffer incrementally decodes byte chunks into Erbsland Core strings. It is useful for file, terminal, and network input where byte chunks can split an encoded code point. It is backed by ByteBuffer and can operate in ordinary or sensitive mode.

The buffer keeps incomplete trailing code points pending until more bytes are written or finish() is called. Use peek...() methods to inspect decoded text without consuming bytes, and take...() methods to consume the bytes that produced the returned complete characters. Use readChar() to consume a single decoded character without constructing a string, and takeStringLine() to consume decoded UTF-8 text up to and including the next LF character.

For the output direction, use StringEncoder::encodedLength() and StringEncoder::encodeTo() with a RingBuffer. The encoder preflights the complete encoded length and atomically writes text into a bounded byte ring without materializing a second byte block.

auto buffer = el::text::StringDecodeBuffer{el::ByteCount{4096}, el::StringEncoding::Utf8};
buffer.write(bytes);
auto text = buffer.takeString();

Unsafe Write Access

text::impl::UnsafeDecodeBufferAccess exposes the first contiguous writable byte span of a StringDecodeBuffer. It is intended for native read APIs that write directly into caller-provided memory. After a successful native read, call commitWritten() with the number of bytes actually written.

Sensitive Decoding

Enable sensitive decoding with setSensitive(true) before writing input bytes. This erases consumed prefixes, malformed sequences, byte-order marks, resets, and the final byte allocation. UTF-8 results are marked sensitive; UTF-16 and UTF-32 results are ordinary because sensitivity is intentionally limited to UTF-8 strings. Disabling sensitive mode securely erases buffered input and resets the decoder.

Base-N Encoding and Decoding

The non-flattened erbsland::text::base_n namespace provides bulk conversion between ByteBlock and all supported string widths. The predefined formats cover RFC 4648 Base16, Base32, Base32hex, Base64, and Base64url. The PEM format factory adds 64-character LF-separated payload wrapping without armor.

Use the explicit namespace because these names are intentionally not added to the flattened erbsland namespace:

namespace base_n = erbsland::text::base_n;

const auto encoded = base_n::BaseNEncoder{data, base_n::BaseNFormat::base64()}.toString();
const auto decoded = base_n::BaseNDecoder{encoded}.toDataOrThrow();

When input bytes are marked sensitive, UTF-8 encoder output is sensitive from allocation through conversion. UTF-16 and UTF-32 encoder output uses ordinary storage because sensitive storage is limited to UTF-8 strings. Decoding a sensitive UTF-8 string creates sensitive byte storage before writing decoded data.

Decoding is strict after configured whitespace is removed. It rejects unknown characters, malformed or missing required padding, incomplete groups, and non-zero unused bits. Use BaseNDecoder::toData() when malformed input and size limit failures should both produce an empty optional. Use BaseNDecoder::toDataOrThrow() when diagnostics are needed: malformed input and decoded-size limits both raise ParseError with a specific reason.

Punycode and IDNA2008

Pure Punycode

The erbsland::text::punycode namespace provides the reversible RFC 3492 Bootstring codec. PunycodeEncoder accepts Unicode String input and produces an ASCII payload, while PunycodeDecoder performs the reverse operation. Default-constructed options select pure Punycode: there is no xn-- prefix handling, domain splitting, normalization, or character policy.

encode() and decode() return an empty optional for data-dependent failures. The OrThrow variants preserve the exact ParseError reason.

Strict IDNA2008

PunycodeOptions can select strict IDNA2008 processing for one label or a complete domain. The network factory folds ASCII uppercase, normalizes Unicode to NFC, validates Unicode 17 derived properties, CONTEXTJ/CONTEXTO and bidi rules, verifies A-label round trips, and enforces DNS byte limits. It deliberately does not apply UTS #46, compatibility or width mappings, Unicode-wide lowercase mapping, or Unicode dot substitutions.

An optional CharSet narrows the canonical Unicode characters accepted after the IDNA2008 checks. Dots remain domain separators and are not tested by that filter.

See Encoding Internationalized Names for practical codec and domain examples.

Capacity and Position Preconditions

Encoded output lengths must fit the supported finite byte length. Exceeding that capacity reports a fatal OutOfRangeFailure. Low-level strict character decoders also report OutOfRangeFailure when called without a character at the supplied read position.

Interface

class BaseNDecoder

Decode Base-N text into binary data.

See: Text Encoding and Conversion

Public Functions

explicit BaseNDecoder(AnyString text, BaseNFormat format = BaseNFormat::defaultFormat()): Create a decoder for text and a format.

inline const BaseNFormat &format() const noexcept: Get the decoding format.

auto toData(unit::ByteCount maximum = unit::ByteCount::infinite()) const -> std::optional<mem::ByteBlock>: Decode data, preserving sensitive input storage and returning no value for invalid input.

mem::ByteBlock toDataOrThrow(unit::ByteCount maximum = unit::ByteCount::infinite()) const

Decode data, preserving sensitive input storage, or throw a detailed error.

Parameters:: maximum – The decoded byte limit, or infinite for no limit. Exceeding this limit is a parse failure.
Throws:: err::ParseError – If the input is malformed or the decoded data exceeds maximum.

class BaseNEncoder

Encode binary data using a validated Base-N format.

See: Text Encoding and Conversion

Public Functions

explicit BaseNEncoder(mem::ByteBlock data, BaseNFormat format = BaseNFormat::defaultFormat()): Create an encoder for data and a format.

inline const BaseNFormat &format() const noexcept: Get the encoding format.

String toString() const: Encode into the primary UTF-8 string type, preserving sensitive input storage.

U8String toU8String() const: Encode into UTF-8, preserving sensitive input storage.

U16String toU16String() const: Encode into UTF-16.

U32String toU32String() const: Encode into UTF-32.

class BaseNFormat

A validated alphabet and behavior definition for power-of-two Base-N encodings.

Supported alphabets contain exactly 16, 32, or 64 distinct Unicode scalar values.

See: Text Encoding and Conversion

Public Functions

BaseNFormat(): Create the canonical Base64 format.

explicit BaseNFormat(U32String alphabet)

Create an unpadded format for a custom alphabet.

ASCII whitespace is accepted while decoding.

Parameters:: alphabet – Exactly 16, 32, or 64 distinct Unicode scalar values. Must not contain ASCII whitespace.

inline const U32String &alphabet() const noexcept: Get the digit alphabet.

BaseNFormat &setAlphabet(U32String alphabet)

Set the digit alphabet.

Parameters:: alphabet – Exactly 16, 32, or 64 distinct Unicode scalar values. Must not overlap the padding or whitespace characters.

inline std::optional<Char> padding() const noexcept: Get the optional padding character.

BaseNFormat &setPadding(std::optional<Char> padding)

Set or clear the padding character.

Parameters:: padding – The optional Unicode scalar padding character. Must not overlap the alphabet or whitespace; required by enabled padding flags.

inline const CharSet &whitespace() const noexcept: Get the characters ignored while decoding.

BaseNFormat &setWhitespace(CharSet whitespace)

Set the characters ignored while decoding.

Parameters:: whitespace – The ignored characters. Must not overlap the alphabet or padding and must include every separator character when wrapping is enabled.

inline BaseNFormatFlags flags() const noexcept: Get the active behavior flags.

BaseNFormat &setFlags(BaseNFormatFlags flags)

Set the behavior flags.

Parameters:: flags – The behavior flags. Padding flags require a padding character; wrapping requires a positive finite line length and a non-empty whitespace-only separator.

BaseNFormat &addFlags(BaseNFormatFlags flags)

Add behavior flags.

Parameters:: flags – The flags to add. Padding flags require a padding character; wrapping requires a positive finite line length and a non-empty whitespace-only separator.

BaseNFormat &clearFlags(BaseNFormatFlags flags)

Clear behavior flags.

Parameters:: flags – The flags to clear. The resulting format must retain valid padding and line-wrapping settings.

inline bool hasFlag(BaseNFormatFlag flag) const noexcept: Test whether a behavior flag is set.

inline unit::CpCount lineLength() const noexcept: Get the encoded digit count per line.

BaseNFormat &setLineLength(unit::CpCount lineLength)

Set the encoded digit count per line.

Parameters:: lineLength – The encoded digit count per line. Must be positive and finite when line wrapping is enabled.

inline const U32String &lineSeparator() const noexcept: Get the separator inserted between encoded lines.

BaseNFormat &setLineSeparator(U32String lineSeparator)

Set the separator inserted between encoded lines.

Parameters:: lineSeparator – The separator between lines. When wrapping is enabled, must be non-empty and consist only of accepted whitespace characters.

uint8_t bitsPerCharacter() const noexcept: Get the number of bits represented by one alphabet character.

std::optional<uint8_t> valueFor(Char character) const noexcept: Find the value represented by a character.

Char characterFor(uint8_t value) const noexcept: Get the alphabet character for a value.

Public Static Functions

static BaseNFormat defaultFormat(): Create the default canonical Base64 format.

static BaseNFormat base16(): Create the canonical RFC 4648 Base16 format.

static BaseNFormat base32(): Create the canonical RFC 4648 Base32 format.

static BaseNFormat base32Hex(): Create the canonical RFC 4648 extended-hex Base32 format.

static BaseNFormat base64(): Create the canonical RFC 4648 Base64 format.

static BaseNFormat base64Url(): Create the canonical RFC 4648 URL-safe Base64 format.

static BaseNFormat base64Pem(): Create padded Base64 output wrapped at 64 characters with LF separators.

enum class erbsland::text::base_n::BaseNFormatFlag : uint8_t

Flags controlling Base-N encoding and decoding.

Values:

enumerator EmitPadding: Emit canonical padding while encoding.

enumerator RequirePadding: Require canonical padding while decoding.

enumerator WrapLines: Wrap encoded output into lines.

enumerator All

using erbsland::text::base_n::BaseNFormatFlags = util::EnumFlags<BaseNFormatFlag>: A set of Base-N format flags.

class EncodingError : public erbsland::err::RuntimeError 

An encoding error exception.

These exceptions are thrown when a text contains encoding errors, and error handing via exception is requested.

Subclassed by erbsland::text::U16EncodingError, erbsland::text::U32EncodingError, erbsland::text::U8EncodingError

Public Functions

inline explicit EncodingError(String reason) noexcept

Create an encoding error exception with a reason.

Parameters:: reason – The reason for the encoding error.

inline explicit EncodingError(const std::string_view reason) noexcept: This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

enum class erbsland::text::EncodingMode : uint8_t

Select how a text conversion handles malformed encoding.

Values:

enumerator Tolerant: Replace invalid input with the Unicode replacement character.

enumerator Strict: Throw a text encoding exception when invalid input is encountered.

class PunycodeDecoder

Decode RFC 3492 Punycode or strict IDNA2008 text into UTF-8.

See: Text Encoding and Conversion

Public Functions

explicit PunycodeDecoder(String text, PunycodeOptions options = {})

Create a decoder.

Parameters:

text – The ASCII input text.
options – The processing options.

std::optional<String> decode() const: Decode the text, returning no value for invalid input.

String decodeOrThrow() const

Decode the text or throw a detailed parse error.

Throws:: err::ParseError – If the input violates Punycode or selected IDNA2008 rules.
Returns:: The decoded UTF-8 text.

inline const PunycodeOptions &options() const noexcept: Get the processing options.

class PunycodeEncoder

Encode Unicode text using RFC 3492 Punycode or strict IDNA2008.

See: Text Encoding and Conversion

Public Functions

explicit PunycodeEncoder(String text, PunycodeOptions options = {})

Create an encoder.

Parameters:

text – The Unicode input text.
options – The processing options.

std::optional<String> encode() const: Encode the text, returning no value for invalid input.

String encodeOrThrow() const

Encode the text or throw a detailed parse error.

Throws:: err::ParseError – If the input violates Punycode or selected IDNA2008 rules.
Returns:: The ASCII Punycode payload, label, or domain.

inline const PunycodeOptions &options() const noexcept: Get the processing options.

enum class erbsland::text::punycode::PunycodeMode : uint8_t

Select pure Punycode or strict IDNA2008 processing.

Values:

enumerator Pure: Apply only the RFC 3492 Bootstring algorithm.

enumerator Idna2008Label: Process one strict IDNA2008 label.

enumerator Idna2008Domain: Process one strict IDNA2008 domain name.

class PunycodeOptions

Options for Punycode and strict IDNA2008 processing.

See: Text Encoding and Conversion

Public Functions

PunycodeOptions() = default: Create pure RFC 3492 options.

inline PunycodeMode mode() const noexcept: Get the processing mode.

inline PunycodeOptions &setMode(const PunycodeMode mode) noexcept: Set the processing mode.

inline bool hasAllowedCharacters() const noexcept: Test whether an additional allowed-character filter is configured.

inline const std::optional<CharSet> &allowedCharacters() const noexcept: Get the additional allowed-character filter.

inline PunycodeOptions &setAllowedCharacters(CharSet allowedCharacters): Set the additional allowed-character filter.

inline PunycodeOptions &clearAllowedCharacters() noexcept: Remove the additional allowed-character filter.

Public Static Functions

static inline PunycodeOptions defaultOptions() noexcept: Create pure RFC 3492 options.

static inline PunycodeOptions idna2008Label() noexcept: Create strict IDNA2008 single-label options.

static inline PunycodeOptions idna2008Domain() noexcept: Create strict IDNA2008 domain-name options.

static inline PunycodeOptions network() noexcept: Create the strict options used at network boundaries.

template<typename T> class StringConverter

Convert between Erbsland Core and standard string types.

Public Types

using Source = T : The source string type.

Public Functions

inline explicit StringConverter(Source source): Create a converter from a source string value.

inline String toString(EncodingMode mode = EncodingMode::Tolerant) const: Convert to the default UTF-8 string type.

inline U8String toU8String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a UTF-8 string.

inline U16String toU16String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a UTF-16 string.

inline U32String toU32String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a UTF-32 string.

inline std::string toStdString(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-8 byte string.

inline std::u8string toStdU8String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-8 string.

inline std::u16string toStdU16String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-16 string.

inline std::u32string toStdU32String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-32 string.

inline std::wstring toStdWString(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard wide string.

class StringDecodeBuffer

A bounded byte buffer for incrementally decoding encoded string data.

Sensitive mode protects discarded input bytes and marks UTF-8 results.

See: Text Encoding and Conversion

Public Types

enum class CodePointStatus : uint8_t

The status of buffered data at the code-point boundary.

Values:

enumerator Complete: The buffer ends at a complete code-point boundary.

enumerator NeedMoreData: A valid trailing prefix needs more bytes.

enumerator Invalid: The buffer contains invalid bytes.

Public Functions

explicit StringDecodeBuffer(unit::ByteCount bufferLength, StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant)

Create a decode buffer.

Parameters:

bufferLength – The fixed byte capacity. Must be at least four bytes.
encoding – The configured text encoding.
bomMode – How byte order marks are handled.
mode – How malformed input is handled.

inline StringEncoding encoding() const noexcept: Get the configured encoding.

inline StringEncoding effectiveEncoding() const noexcept: Get the effective encoding after BOM resolution.

inline unit::ByteCount capacity() const noexcept: Get the byte capacity.

inline unit::ByteCount availableSpace() const noexcept: Get the available byte space.

inline unit::ByteCount byteLength() const noexcept: Get the number of buffered bytes.

unit::CpCount decodableCharacters(unit::CpCount maximum = unit::CpCount::infinite()): Count complete characters available for decoding.

inline bool isEmpty() const noexcept: Test if no bytes are buffered.

inline bool isFinished() const noexcept: Test if input was marked complete.

CodePointStatus codePointStatus(): Get the code-point boundary status.

inline bool isCodePointComplete(): Test if buffered data ends at a complete code-point boundary.

void write(mem::ConstByteSpan bytes): Write borrowed bytes into available storage.

void write(const mem::ByteBlock &bytes): Write an ordinary byte block.

void write(const std::vector<mem::Byte> &bytes): Write explicit byte values.

void write(const std::vector<uint8_t> &bytes): Write unsigned byte values.

void write(const std::vector<char> &bytes): Write character byte values.

void write(std::string_view bytes): Write character bytes.

void writeStringBytes(const String &bytes): Write raw UTF-8 bytes from a string.

inline void finish() noexcept: Mark input complete.

void reset() noexcept: Reset decoder state, preserving its sensitivity mode.

inline bool isSensitive() const noexcept: Test if discarded decoder storage is securely erased and UTF-8 results are marked.

void setSensitive(bool sensitive) noexcept

Enable or disable sensitive decoder storage.

Disabling securely erases buffered input and resets decoder state.

AnyString peekAnyString(unit::CpCount maximum = unit::CpCount::infinite()): Decode available bytes to a string matching the effective encoding.

String peekString(unit::CpCount maximum = unit::CpCount::infinite()): Decode available bytes to the default string type.

U8String peekU8String(unit::CpCount maximum = unit::CpCount::infinite()): Decode available bytes to a UTF-8 string.

U16String peekU16String(unit::CpCount maximum = unit::CpCount::infinite()): Decode available bytes to a UTF-16 string.

U32String peekU32String(unit::CpCount maximum = unit::CpCount::infinite()): Decode available bytes to a UTF-32 string.

std::optional<Char> readChar(): Decode and consume one character.

AnyString takeAnyString(unit::CpCount maximum = unit::CpCount::infinite()): Decode and consume available bytes to a string matching the effective encoding.

String takeString(unit::CpCount maximum = unit::CpCount::infinite()): Decode and consume available bytes to the default string type.

String takeStringLine(unit::CpCount maximum = unit::CpCount::infinite()): Decode and consume one line to the default string type.

U8String takeU8String(unit::CpCount maximum = unit::CpCount::infinite()): Decode and consume available bytes to a UTF-8 string.

U16String takeU16String(unit::CpCount maximum = unit::CpCount::infinite()): Decode and consume available bytes to a UTF-16 string.

U32String takeU32String(unit::CpCount maximum = unit::CpCount::infinite()): Decode and consume available bytes to a UTF-32 string.

class StringDecoder

Decode binary text data into Erbsland Core strings.

Public Functions

StringDecoder(const mem::ByteBlockEditor &data) noexcept: Create a decoder sharing data from a byte block editor.

StringDecoder(const mem::ByteBlock &data) noexcept: Create a decoder sharing data from a read-only byte block.

void validateOrThrow(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const

Validate that the byte data is strictly encoded text without creating a string.

Parameters:

encoding – The expected encoding.
bomMode – How an initial byte order mark is handled.

Throws:

EncodingError – If the byte order mark or encoded text is invalid.

auto decode(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> String 

Decode to the default UTF-8 string type.

Tolerant UTF-8 input preserves the payload bytes after BOM handling.

auto toU8String(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> U8String 

Decode to a UTF-8 string.

Tolerant UTF-8 input preserves the payload bytes after BOM handling.

auto toU16String(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> U16String : Decode to a UTF-16 string.

auto toU32String(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> U32String : Decode to a UTF-32 string.

template<impl::AnyStringOrStringEditorType T> class StringEncoder

Encode Erbsland Core strings as binary text data.

Public Types

using Source = T : The source string type.

Public Functions

inline explicit StringEncoder(const Source &source) noexcept: Create an encoder for the given source string.

auto encode(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const -> mem::ByteBlock 

Encode the source string into the requested byte encoding.

Matching native representations are copied without validation. Actual transcoding replaces malformed source sequences.

auto encodedLength(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const -> unit::ByteCount 

Calculate the exact byte length produced by encode() without allocating the encoded byte block.

Matching native representations are measured without validation. The encoded length must fit the supported finite byte length.

auto encodeTo(mem::RingBuffer &buffer, StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const -> util::Result 

Atomically encode the source directly into a ring buffer.

Each call independently applies bomMode; stateful streams must suppress the BOM after their first write. Matching native representations are copied without validation. Actual transcoding replaces malformed source sequences.

Returns:: Success, or failure without modification if the encoded data exceeds the remaining hard capacity. The encoded length must fit the supported finite byte length.

class StringEncoding

A binary encoding supported by string byte conversion APIs.

Multi-byte encodings use either the explicit byte order named by the value, or use a byte order mark to indicate the byte order when decoding. Generic UTF-16 and UTF-32 encode as little-endian.

See: Text Encoding and Conversion

Public Types

enum Value

The raw string encoding value.

Values:

enumerator Utf8: UTF-8 bytes without a byte order mark by default.

enumerator Utf16: UTF-16 bytes, detect byte order from byte order mark, encode as little-endian.

enumerator Utf16LittleEndian: UTF-16 bytes, least significant byte first.

enumerator Utf16BigEndian: UTF-16 bytes, most significant byte first.

enumerator Utf32: UTF-32 bytes, detect byte order from byte order mark, encode as little-endian.

enumerator Utf32LittleEndian: UTF-32 bytes, least significant byte first.

enumerator Utf32BigEndian: UTF-32 bytes, most significant byte first.

Public Functions

inline constexpr StringEncoding(const Value value) noexcept: Create a string encoding from a raw value.

inline constexpr Value toRawValue() const noexcept: Get the raw string encoding value.

inline constexpr bool isUtf8() const noexcept: Test if this is the UTF-8 encoding.

inline constexpr bool isUtf16() const noexcept: Test if this is a generic or byte-order-specific UTF-16 encoding.

inline constexpr bool isUtf32() const noexcept: Test if this is a generic or byte-order-specific UTF-32 encoding.

inline constexpr StringEncoding effectiveEncoding() const noexcept: Resolve a generic encoding to the concrete encoding used for output.

inline constexpr mem::Endianness endianness() const noexcept

Get the byte order used for output.

UTF-8 reports little-endian as the library default, although byte order has no effect on UTF-8 data.

inline constexpr bool writesBom(const StringBomMode mode) const noexcept: Test if this encoding writes a byte order mark in the given mode.

unit::ByteCount bomLength(StringBomMode mode) const noexcept: Get the byte length of the byte order mark written in the given mode.

mem::ConstByteSpan bomBytes(StringBomMode mode) const noexcept

Get the byte order mark written in the given mode.

The returned span is empty if this encoding does not write a byte order mark in the selected mode.

enum class erbsland::text::StringKind : uint8_t

The string encoding kind used by generic text APIs.

The indexes of this enum are used as type indices for std::variant.

Values:

enumerator U8

enumerator U16

enumerator U32

String erbsland::text::toString(StringKind kind): Convert a string kind to its descriptive text.

String erbsland::text::toString(const String &value): Convert a string to the common string type.

String erbsland::text::toString(bool value, BooleanFormat format = BooleanFormat::defaultFormat()): Convert a boolean value to a string.

String erbsland::text::toString(std::strong_ordering value): Convert a strong ordering value to a string.

String erbsland::text::toString(int8_t value, IntegerFormat format = IntegerFormat::defaultFormat()): Convert an integer/float value to a string.

String erbsland::text::toString(int16_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(int32_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(int64_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint8_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint16_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint32_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint64_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(float value, FloatFormat format = FloatFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(double value, FloatFormat format = FloatFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

class U16EncodingError : public erbsland::text::EncodingError 

A UTF-16 encoding error exception.

Public Functions

inline explicit U16EncodingError(const std::string_view reason, const unit::U16Index index) noexcept

Create a UTF-16 encoding error exception with a reason.

Parameters:

reason – The reason for the encoding error.
index – The word index where the error occurred.

inline unit::U16Index index() const noexcept: Get the UTF-16 data index where the malformed sequence was detected.

class U32EncodingError : public erbsland::text::EncodingError 

A UTF-32 encoding error exception.

Public Functions

inline explicit U32EncodingError(const std::string_view reason, const unit::CpIndex index) noexcept

Create a UTF-32 encoding error exception with a reason.

Parameters:

reason – The reason for the encoding error.
index – The code-point index where the error occurred.

inline unit::CpIndex index() const noexcept: Get the code-point index where the invalid value was detected.

class U8EncodingError : public erbsland::text::EncodingError 

A UTF-8 encoding error exception.

Public Functions

inline explicit U8EncodingError(const std::string_view reason, const unit::ByteIndex index) noexcept

Create a UTF-8 encoding error exception with a reason.

Parameters:

reason – The reason for the encoding error.
index – The byte index where the error occurred.

inline unit::ByteIndex index() const noexcept: Get the byte index where the malformed sequence was detected.