Text Encoding and Conversion

String Converter

Introduction

StringConverter is the explicit conversion entry point for Erbsland Core strings, editors, literals and the supported standard-library string types. It keeps conversion helpers out of the core string classes while still allowing concise call sites.

auto text = el::StringConverter{std::string_view{"Hello"}}.toString();
auto utf16 = el::StringConverter{text}.toStdU16String();
auto converted = el::StringConverter{utf16}.toString();

toString() returns a read-only UTF-8 value. It aliases compatible Core storage in tolerant mode and owns newly converted storage. Every conversion method accepts EncodingMode, defaulting to Tolerant.

Extension libraries can support additional source types by specializing StringConverterTraits.

Encoding Mode

EncodingMode controls how encoding errors are handled by StringConverter. Tolerant is the default and replaces malformed input with Unicode replacement characters when transcoding. Compatible representations may instead be copied unchanged without validation. Strict throws EncodingError when malformed input is encountered. Use strict mode when a conversion must also validate its source, including an existing Erbsland Core string.

String Decoder

StringDecoder decodes ByteBlock or ByteBlockEditor input into Erbsland Core strings.

auto text = el::StringDecoder{bytes}.decode(el::StringEncoding::Utf8);
auto utf16 = el::StringDecoder{bytes}.toU16String(el::StringEncoding::Utf16);
el::StringDecoder{bytes}.validateOrThrow(el::StringEncoding::Utf8);

The decoder accepts StringEncoding, StringBomMode, and EncodingMode for all target string widths. Only an encoded U+FEFF signature at the start of the byte input is interpreted as a BOM. A repeated or embedded U+FEFF is invalid content and follows EncodingMode; it is never returned as a character in the decoded string.

Use validateOrThrow() when encoded bytes must be checked without constructing a decoded string. It applies strict validation together with the selected encoding and BOM policy, traverses the input once, and allocates no output storage.

String Encoder

StringEncoder encodes Erbsland Core strings and editors into ByteBlock data or directly into a RingBuffer. Use encodedLength() to calculate the exact byte length without allocating an intermediate byte block. The encodeTo() method reserves the complete output and commits it atomically; insufficient capacity leaves readable ring data unchanged. The encoder is intentionally limited to Erbsland Core text sources.

auto bytes = el::StringEncoder{text}.encode(el::StringEncoding::Utf8, el::StringBomMode::Reject);
auto byteLength = el::StringEncoder{text}.encodedLength(el::StringEncoding::Utf16);
auto result = el::StringEncoder{text}.encodeTo(ring, el::StringEncoding::Utf16);

Each standalone encode() or encodeTo() call applies its requested BOM policy independently, including calls for empty text that produce only a BOM. Stateful output streams suppress the BOM after their first successful write. Explicit BOM output is generated as an encoding signature. When the source and target representations match, the encoder copies the native units directly without a validation pass because Erbsland Core strings are assumed to contain valid text. When transcoding is necessary, malformed source sequences become Unicode replacement characters. Call the matching isValidUtf8(), isValidUtf16(), or isValidUtf32() method before encoding when invalid internal text must be detected. An ordinary Char{0xFEFF} in source content is never interpreted as a signature; during transcoding it becomes a replacement character like other invalid content.

StringEncoder accepts the read-only string and editor types for all three supported string widths.

String Kind

StringKind selects the concrete string encoding used by generic text construction APIs. Use it when user code should decide whether a result is built as UTF-8, UTF-16, or UTF-32 text.

Usage

Pass StringKind to AnyStringBuilder when you need an empty builder for a specific output encoding.

auto utf8 = el::AnyStringBuilder{el::StringKind::U8};
auto utf16 = el::AnyStringBuilder{el::StringKind::U16};
auto utf32 = el::AnyStringBuilder{el::StringKind::U32};

Any-Width String Values

AnyString stores a read-only UTF-8, UTF-16 or UTF-32 string without changing its width. It can be compared directly with another AnyString, a width-specific Core string or editor, or any _el string literal. Comparisons are lexicographical by decoded code point and do not create converted strings. Malformed encoded units are compared as Char::replacement().

auto pattern = el::AnyString{u"[a-z]+"_el};

if (pattern == "[a-z]+"_el) {
    // The UTF-16 pattern is compared directly with the UTF-8 literal.
}

An empty AnyString has no selected width. It compares equal to empty strings of every width and sorts before any non-empty string.

Standalone Conversion Helpers

The toString() overloads convert common text, numeric, boolean, and ordering values into the common String type. Boolean conversion accepts BooleanFormat. The std::strong_ordering overload returns less, equal, or greater.

String Decode Buffer

Introduction

StringDecodeBuffer incrementally decodes byte chunks into Erbsland Core strings. It is useful for file, terminal, and network input where byte chunks can split an encoded code point. It is backed by ByteBuffer and can operate in ordinary or sensitive mode.

The buffer keeps incomplete trailing code points pending until more bytes are written or finish() is called. Use peek...() methods to inspect decoded text without consuming bytes, and take...() methods to consume the bytes that produced the returned complete characters. Use readChar() to consume a single decoded character without constructing a string, and takeStringLine() to consume decoded UTF-8 text up to and including the next LF character.

For the output direction, use StringEncoder::encodedLength() and StringEncoder::encodeTo() with a RingBuffer. The encoder preflights the complete encoded length and atomically writes text into a bounded byte ring without materializing a second byte block.

auto buffer = el::text::StringDecodeBuffer{el::ByteLength{4096}, el::StringEncoding::Utf8};
buffer.write(bytes);
auto text = buffer.takeString();

Unsafe Write Access

text::impl::UnsafeDecodeBufferAccess exposes the first contiguous writable byte span of a StringDecodeBuffer. It is intended for native read APIs that write directly into caller-provided memory. After a successful native read, call commitWritten() with the number of bytes actually written.

Sensitive Decoding

Enable sensitive decoding with setSensitive(true) before writing input bytes. This erases consumed prefixes, malformed sequences, byte-order marks, resets, and the final byte allocation. UTF-8 results are marked sensitive; UTF-16 and UTF-32 results are ordinary because sensitivity is intentionally limited to UTF-8 strings. Disabling sensitive mode securely erases buffered input and resets the decoder.

Base-N Encoding and Decoding

The non-flattened erbsland::text::base_n namespace provides bulk conversion between ByteBlock and all supported string widths. The predefined formats cover RFC 4648 Base16, Base32, Base32hex, Base64, and Base64url. The PEM format factory adds 64-character LF-separated payload wrapping without armor.

Use the explicit namespace because these names are intentionally not added to the flattened erbsland namespace:

namespace base_n = erbsland::text::base_n;

const auto encoded = base_n::BaseNEncoder{data, base_n::BaseNFormat::base64()}.toString();
const auto decoded = base_n::BaseNDecoder{encoded}.toDataOrThrow();

Decoding is strict after configured whitespace is removed. It rejects unknown characters, malformed or missing required padding, incomplete groups, and non-zero unused bits. Use BaseNDecoder::toData() when malformed input and size limit failures should both produce an empty optional. Use BaseNDecoder::toDataOrThrow() when diagnostics must distinguish parse failures from decoded-size limits.

Punycode and IDNA2008

Pure Punycode

The erbsland::text::punycode namespace provides the reversible RFC 3492 Bootstring codec. PunycodeEncoder accepts Unicode String input and produces an ASCII payload, while PunycodeDecoder performs the reverse operation. Default-constructed options select pure Punycode: there is no xn-- prefix handling, domain splitting, normalization, or character policy.

encode() and decode() return an empty optional for data-dependent failures. The OrThrow variants preserve the exact ParseError reason.

Strict IDNA2008

PunycodeOptions can select strict IDNA2008 processing for one label or a complete domain. The network factory folds ASCII uppercase, normalizes Unicode to NFC, validates Unicode 17 derived properties, CONTEXTJ/CONTEXTO and bidi rules, verifies A-label round trips, and enforces DNS byte limits. It deliberately does not apply UTS #46, compatibility or width mappings, Unicode-wide lowercase mapping, or Unicode dot substitutions.

An optional CharSet narrows the canonical Unicode characters accepted after the IDNA2008 checks. Dots remain domain separators and are not tested by that filter.

See Encoding Internationalized Names for practical codec and domain examples.

Interface

class BaseNDecoder

Decode Base-N text into binary data.

See: Text Encoding and Conversion

Public Functions

explicit BaseNDecoder(AnyString text, BaseNFormat format = BaseNFormat::defaultFormat()): Create a decoder for text and a format.

inline const BaseNFormat &format() const noexcept: Get the decoding format.

std::optional<mem::ByteBlock> toData(unit::ByteLength maximum = unit::ByteLength::infinite()) const: Decode data, returning no value for malformed text or a size-limit violation.

mem::ByteBlock toDataOrThrow(unit::ByteLength maximum = unit::ByteLength::infinite()) const

Decode data or throw a detailed error.

Throws:

err::ParseError – If the text is malformed.
err::OutOfRangeError – If the decoded data exceeds maximum.

class BaseNEncoder

Encode binary data using a validated Base-N format.

See: Text Encoding and Conversion

Public Functions

explicit BaseNEncoder(mem::ByteBlock data, BaseNFormat format = BaseNFormat::defaultFormat()): Create an encoder for data and a format.

inline const BaseNFormat &format() const noexcept: Get the encoding format.

String toString() const: Encode into the primary UTF-8 string type.

U8String toU8String() const: Encode into UTF-8.

U16String toU16String() const: Encode into UTF-16.

U32String toU32String() const: Encode into UTF-32.

class BaseNFormat

A validated alphabet and behavior definition for power-of-two Base-N encodings.

Supported alphabets contain exactly 16, 32, or 64 distinct Unicode scalar values.

See: Text Encoding and Conversion

Public Functions

BaseNFormat(): Create the canonical Base64 format.

explicit BaseNFormat(U32String alphabet)

Create an unpadded format for a custom alphabet.

ASCII whitespace is accepted while decoding.

Parameters:: alphabet – An alphabet containing exactly 16, 32, or 64 distinct Unicode scalar values.
Throws:: err::ParameterError – If the alphabet is invalid.

inline const U32String &alphabet() const noexcept: Get the digit alphabet.

BaseNFormat &setAlphabet(U32String alphabet)

Set the digit alphabet.

Throws:: err::ParameterError – If the resulting format is invalid.

inline std::optional<Char> padding() const noexcept: Get the optional padding character.

BaseNFormat &setPadding(std::optional<Char> padding)

Set or clear the padding character.

Throws:: err::ParameterError – If the resulting format is invalid.

inline const CharSet &whitespace() const noexcept: Get the characters ignored while decoding.

BaseNFormat &setWhitespace(CharSet whitespace)

Set the characters ignored while decoding.

Throws:: err::ParameterError – If the resulting format is invalid.

inline BaseNFormatFlags flags() const noexcept: Get the active behavior flags.

BaseNFormat &setFlags(BaseNFormatFlags flags)

Set the behavior flags.

Throws:: err::ParameterError – If the resulting format is invalid.

BaseNFormat &addFlags(BaseNFormatFlags flags)

Add behavior flags.

Throws:: err::ParameterError – If the resulting format is invalid.

BaseNFormat &clearFlags(BaseNFormatFlags flags)

Clear behavior flags.

Throws:: err::ParameterError – If the resulting format is invalid.

inline bool hasFlag(BaseNFormatFlag flag) const noexcept: Test whether a behavior flag is set.

inline unit::CpLength lineLength() const noexcept: Get the encoded digit count per line.

BaseNFormat &setLineLength(unit::CpLength lineLength)

Set the encoded digit count per line.

Throws:: err::ParameterError – If the resulting format is invalid.

inline const U32String &lineSeparator() const noexcept: Get the separator inserted between encoded lines.

BaseNFormat &setLineSeparator(U32String lineSeparator)

Set the separator inserted between encoded lines.

Throws:: err::ParameterError – If the resulting format is invalid.

uint8_t bitsPerCharacter() const noexcept: Get the number of bits represented by one alphabet character.

std::optional<uint8_t> valueFor(Char character) const noexcept: Find the value represented by a character.

Char characterFor(uint8_t value) const noexcept: Get the alphabet character for a value.

Public Static Functions

static BaseNFormat defaultFormat(): Create the default canonical Base64 format.

static BaseNFormat base16(): Create the canonical RFC 4648 Base16 format.

static BaseNFormat base32(): Create the canonical RFC 4648 Base32 format.

static BaseNFormat base32Hex(): Create the canonical RFC 4648 extended-hex Base32 format.

static BaseNFormat base64(): Create the canonical RFC 4648 Base64 format.

static BaseNFormat base64Url(): Create the canonical RFC 4648 URL-safe Base64 format.

static BaseNFormat base64Pem(): Create padded Base64 output wrapped at 64 characters with LF separators.

enum class erbsland::text::base_n::BaseNFormatFlag : uint8_t

Flags controlling Base-N encoding and decoding.

Values:

enumerator EmitPadding: Emit canonical padding while encoding.

enumerator RequirePadding: Require canonical padding while decoding.

enumerator WrapLines: Wrap encoded output into lines.

enumerator All

using erbsland::text::base_n::BaseNFormatFlags = util::EnumFlags<BaseNFormatFlag>: A set of Base-N format flags.

class EncodingError : public erbsland::err::RuntimeError 

An encoding error exception.

These exceptions are thrown when a text contains encoding errors, and error handing via exception is requested.

Subclassed by erbsland::text::U16EncodingError, erbsland::text::U32EncodingError, erbsland::text::U8EncodingError

Public Functions

inline explicit EncodingError(String reason) noexcept

Create an encoding error exception with a reason.

Parameters:: reason – The reason for the encoding error.

inline explicit EncodingError(const std::string_view reason) noexcept: This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

enum class erbsland::text::EncodingMode : uint8_t

Select how a text conversion handles malformed encoding.

Values:

enumerator Tolerant: Replace invalid input with the Unicode replacement character.

enumerator Strict: Throw a text encoding exception when invalid input is encountered.

class PunycodeDecoder

Decode RFC 3492 Punycode or strict IDNA2008 text into UTF-8.

See: Text Encoding and Conversion

Public Functions

explicit PunycodeDecoder(String text, PunycodeOptions options = {})

Create a decoder.

Parameters:

text – The ASCII input text.
options – The processing options.

std::optional<String> decode() const: Decode the text, returning no value for invalid input.

String decodeOrThrow() const

Decode the text or throw a detailed parse error.

Throws:: err::ParseError – If the input violates Punycode or selected IDNA2008 rules.
Returns:: The decoded UTF-8 text.

inline const PunycodeOptions &options() const noexcept: Get the processing options.

class PunycodeEncoder

Encode Unicode text using RFC 3492 Punycode or strict IDNA2008.

See: Text Encoding and Conversion

Public Functions

explicit PunycodeEncoder(String text, PunycodeOptions options = {})

Create an encoder.

Parameters:

text – The Unicode input text.
options – The processing options.

std::optional<String> encode() const: Encode the text, returning no value for invalid input.

String encodeOrThrow() const

Encode the text or throw a detailed parse error.

Throws:: err::ParseError – If the input violates Punycode or selected IDNA2008 rules.
Returns:: The ASCII Punycode payload, label, or domain.

inline const PunycodeOptions &options() const noexcept: Get the processing options.

enum class erbsland::text::punycode::PunycodeMode : uint8_t

Select pure Punycode or strict IDNA2008 processing.

Values:

enumerator Pure: Apply only the RFC 3492 Bootstring algorithm.

enumerator Idna2008Label: Process one strict IDNA2008 label.

enumerator Idna2008Domain: Process one strict IDNA2008 domain name.

class PunycodeOptions

Options for Punycode and strict IDNA2008 processing.

See: Text Encoding and Conversion

Public Functions

PunycodeOptions() = default: Create pure RFC 3492 options.

inline PunycodeMode mode() const noexcept: Get the processing mode.

inline PunycodeOptions &setMode(const PunycodeMode mode) noexcept: Set the processing mode.

inline bool hasAllowedCharacters() const noexcept: Test whether an additional allowed-character filter is configured.

inline const std::optional<CharSet> &allowedCharacters() const noexcept: Get the additional allowed-character filter.

inline PunycodeOptions &setAllowedCharacters(CharSet allowedCharacters): Set the additional allowed-character filter.

inline PunycodeOptions &clearAllowedCharacters() noexcept: Remove the additional allowed-character filter.

Public Static Functions

static inline PunycodeOptions defaultOptions() noexcept: Create pure RFC 3492 options.

static inline PunycodeOptions idna2008Label() noexcept: Create strict IDNA2008 single-label options.

static inline PunycodeOptions idna2008Domain() noexcept: Create strict IDNA2008 domain-name options.

static inline PunycodeOptions network() noexcept: Create the strict options used at network boundaries.

template<typename T> class StringConverter

Convert between Erbsland Core and standard string types.

Public Types

using Source = T : The source string type.

Public Functions

inline explicit StringConverter(Source source): Create a converter from a source string value.

inline String toString(EncodingMode mode = EncodingMode::Tolerant) const: Convert to the default UTF-8 string type.

inline U8String toU8String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a UTF-8 string.

inline U16String toU16String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a UTF-16 string.

inline U32String toU32String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a UTF-32 string.

inline std::string toStdString(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-8 byte string.

inline std::u8string toStdU8String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-8 string.

inline std::u16string toStdU16String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-16 string.

inline std::u32string toStdU32String(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard UTF-32 string.

inline std::wstring toStdWString(EncodingMode mode = EncodingMode::Tolerant) const: Convert to a standard wide string.

class StringDecodeBuffer

A bounded byte buffer for incrementally decoding encoded string data.

Sensitive mode protects discarded input bytes and marks UTF-8 results.

See: Text Encoding and Conversion

Public Types

enum class CodePointStatus : uint8_t

The status of buffered data at the code-point boundary.

Values:

enumerator Complete: The buffer ends at a complete code-point boundary.

enumerator NeedMoreData: A valid trailing prefix needs more bytes.

enumerator Invalid: The buffer contains invalid bytes.

Public Functions

explicit StringDecodeBuffer(unit::ByteLength bufferLength, StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant)

Create a decode buffer.

Parameters:

bufferLength – The fixed byte capacity. Must be at least four bytes.
encoding – The configured text encoding.
bomMode – How byte order marks are handled.
mode – How malformed input is handled.

Throws:

err::ParameterError – If bufferLength is smaller than four bytes.

inline StringEncoding encoding() const noexcept: Get the configured encoding.

inline StringEncoding effectiveEncoding() const noexcept: Get the effective encoding after BOM resolution.

inline unit::ByteLength capacity() const noexcept: Get the byte capacity.

inline unit::ByteLength availableSpace() const noexcept: Get the available byte space.

inline unit::ByteLength byteLength() const noexcept: Get the number of buffered bytes.

unit::CpLength decodableCharacters(unit::CpLength maximum = unit::CpLength::infinite()): Count complete characters available for decoding.

inline bool isEmpty() const noexcept: Test if no bytes are buffered.

inline bool isFinished() const noexcept: Test if input was marked complete.

CodePointStatus codePointStatus(): Get the code-point boundary status.

inline bool isCodePointComplete(): Test if buffered data ends at a complete code-point boundary.

void write(mem::ConstByteSpan bytes): Write borrowed bytes into available storage.

void write(const mem::ByteBlock &bytes): Write an ordinary byte block.

void write(const std::vector<mem::Byte> &bytes): Write explicit byte values.

void write(const std::vector<uint8_t> &bytes): Write unsigned byte values.

void write(const std::vector<char> &bytes): Write character byte values.

void write(std::string_view bytes): Write character bytes.

void writeStringBytes(const String &bytes): Write raw UTF-8 bytes from a string.

inline void finish() noexcept: Mark input complete.

void reset() noexcept: Reset decoder state, preserving its sensitivity mode.

inline bool isSensitive() const noexcept: Test if discarded decoder storage is securely erased and UTF-8 results are marked.

void setSensitive(bool sensitive) noexcept

Enable or disable sensitive decoder storage.

Disabling securely erases buffered input and resets decoder state.

AnyString peekAnyString(unit::CpLength maximum = unit::CpLength::infinite()): Decode available bytes to a string matching the effective encoding.

String peekString(unit::CpLength maximum = unit::CpLength::infinite()): Decode available bytes to the default string type.

U8String peekU8String(unit::CpLength maximum = unit::CpLength::infinite()): Decode available bytes to a UTF-8 string.

U16String peekU16String(unit::CpLength maximum = unit::CpLength::infinite()): Decode available bytes to a UTF-16 string.

U32String peekU32String(unit::CpLength maximum = unit::CpLength::infinite()): Decode available bytes to a UTF-32 string.

std::optional<Char> readChar(): Decode and consume one character.

AnyString takeAnyString(unit::CpLength maximum = unit::CpLength::infinite()): Decode and consume available bytes to a string matching the effective encoding.

String takeString(unit::CpLength maximum = unit::CpLength::infinite()): Decode and consume available bytes to the default string type.

String takeStringLine(unit::CpLength maximum = unit::CpLength::infinite()): Decode and consume one line to the default string type.

U8String takeU8String(unit::CpLength maximum = unit::CpLength::infinite()): Decode and consume available bytes to a UTF-8 string.

U16String takeU16String(unit::CpLength maximum = unit::CpLength::infinite()): Decode and consume available bytes to a UTF-16 string.

U32String takeU32String(unit::CpLength maximum = unit::CpLength::infinite()): Decode and consume available bytes to a UTF-32 string.

class StringDecoder

Decode binary text data into Erbsland Core strings.

Public Functions

StringDecoder(const mem::ByteBlockEditor &data) noexcept: Create a decoder sharing data from a byte block editor.

StringDecoder(const mem::ByteBlock &data) noexcept: Create a decoder sharing data from a read-only byte block.

void validateOrThrow(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const

Validate that the byte data is strictly encoded text without creating a string.

Parameters:

encoding – The expected encoding.
bomMode – How an initial byte order mark is handled.

Throws:

EncodingError – If the byte order mark or encoded text is invalid.

auto decode(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> String : Decode to the default UTF-8 string type.

auto toU8String(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> U8String : Decode to a UTF-8 string.

auto toU16String(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> U16String : Decode to a UTF-16 string.

auto toU32String(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic, EncodingMode mode = EncodingMode::Tolerant) const -> U32String : Decode to a UTF-32 string.

template<impl::AnyStringOrStringEditorType T> class StringEncoder

Encode Erbsland Core strings as binary text data.

Public Types

using Source = T : The source string type.

Public Functions

inline explicit StringEncoder(const Source &source) noexcept: Create an encoder for the given source string.

mem::ByteBlock encode(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const

Encode the source string into the requested byte encoding.

Matching native representations are copied without validation. Actual transcoding replaces malformed source sequences.

unit::ByteLength encodedLength(StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const

Calculate the exact byte length produced by encode() without allocating the encoded byte block.

Matching native representations are measured without validation.

Throws:: err::OverflowError – If the encoded length exceeds the supported finite byte length.

auto encodeTo(mem::RingBuffer &buffer, StringEncoding encoding, StringBomMode bomMode = StringBomMode::Automatic) const -> util::Result 

Atomically encode the source directly into a ring buffer.

Each call independently applies bomMode; stateful streams must suppress the BOM after their first write. Matching native representations are copied without validation. Actual transcoding replaces malformed source sequences.

Throws:: err::OverflowError – If the encoded length exceeds the supported finite byte length.
Returns:: Success, or failure without modification if the encoded data exceeds the remaining hard capacity.

class StringEncoding

A binary encoding supported by string byte conversion APIs.

Multi-byte encodings use either the explicit byte order named by the value, or use a byte order mark to indicate the byte order when decoding. Generic UTF-16 and UTF-32 encode as little-endian.

See: Text Encoding and Conversion

Public Types

enum Value

The raw string encoding value.

Values:

enumerator Utf8: UTF-8 bytes without a byte order mark by default.

enumerator Utf16: UTF-16 bytes, detect byte order from byte order mark, encode as little-endian.

enumerator Utf16LittleEndian: UTF-16 bytes, least significant byte first.

enumerator Utf16BigEndian: UTF-16 bytes, most significant byte first.

enumerator Utf32: UTF-32 bytes, detect byte order from byte order mark, encode as little-endian.

enumerator Utf32LittleEndian: UTF-32 bytes, least significant byte first.

enumerator Utf32BigEndian: UTF-32 bytes, most significant byte first.

Public Functions

inline constexpr StringEncoding(const Value value) noexcept: Create a string encoding from a raw value.

inline constexpr Value toRawValue() const noexcept: Get the raw string encoding value.

inline constexpr bool isUtf8() const noexcept: Test if this is the UTF-8 encoding.

inline constexpr bool isUtf16() const noexcept: Test if this is a generic or byte-order-specific UTF-16 encoding.

inline constexpr bool isUtf32() const noexcept: Test if this is a generic or byte-order-specific UTF-32 encoding.

inline constexpr StringEncoding effectiveEncoding() const noexcept: Resolve a generic encoding to the concrete encoding used for output.

inline constexpr mem::Endianness endianness() const noexcept

Get the byte order used for output.

UTF-8 reports little-endian as the library default, although byte order has no effect on UTF-8 data.

inline constexpr bool writesBom(const StringBomMode mode) const noexcept: Test if this encoding writes a byte order mark in the given mode.

unit::ByteLength bomLength(StringBomMode mode) const noexcept: Get the byte length of the byte order mark written in the given mode.

mem::ConstByteSpan bomBytes(StringBomMode mode) const noexcept

Get the byte order mark written in the given mode.

The returned span is empty if this encoding does not write a byte order mark in the selected mode.

enum class erbsland::text::StringKind : uint8_t

The string encoding kind used by generic text APIs.

The indexes of this enum are used as type indices for std::variant.

Values:

enumerator U8

enumerator U16

enumerator U32

String erbsland::text::toString(StringKind kind): Convert a string kind to its descriptive text.

String erbsland::text::toString(const String &value): Convert a string to the common string type.

String erbsland::text::toString(bool value, BooleanFormat format = BooleanFormat::defaultFormat()): Convert a boolean value to a string.

String erbsland::text::toString(std::strong_ordering value): Convert a strong ordering value to a string.

String erbsland::text::toString(int8_t value, IntegerFormat format = IntegerFormat::defaultFormat()): Convert an integer/float value to a string.

String erbsland::text::toString(int16_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(int32_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(int64_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint8_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint16_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint32_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(uint64_t value, IntegerFormat format = IntegerFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(float value, FloatFormat format = FloatFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

String erbsland::text::toString(double value, FloatFormat format = FloatFormat::defaultFormat()): This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

class U16EncodingError : public erbsland::text::EncodingError 

A UTF-16 encoding error exception.

Public Functions

inline explicit U16EncodingError(const std::string_view reason, const unit::U16DataIndex index) noexcept

Create a UTF-16 encoding error exception with a reason.

Parameters:

reason – The reason for the encoding error.
index – The word index where the error occurred.

inline unit::U16DataIndex index() const noexcept: Get the UTF-16 data index where the malformed sequence was detected.

class U32EncodingError : public erbsland::text::EncodingError 

A UTF-32 encoding error exception.

Public Functions

inline explicit U32EncodingError(const std::string_view reason, const unit::CpIndex index) noexcept

Create a UTF-32 encoding error exception with a reason.

Parameters:

reason – The reason for the encoding error.
index – The code-point index where the error occurred.

inline unit::CpIndex index() const noexcept: Get the code-point index where the invalid value was detected.

class U8EncodingError : public erbsland::text::EncodingError 

A UTF-8 encoding error exception.

Public Functions

inline explicit U8EncodingError(const std::string_view reason, const unit::ByteIndex index) noexcept

Create a UTF-8 encoding error exception with a reason.

Parameters:

reason – The reason for the encoding error.
index – The byte index where the error occurred.

inline unit::ByteIndex index() const noexcept: Get the byte index where the malformed sequence was detected.