Characters and Unicode
Char Range
Introduction
Case Sensitivity
CaseSensitivity selects exact or case-insensitive character-wise comparison for string comparison, search, and
replacement APIs.
Use comparisonFn() for Unicode simple case folding.
Use asciiComparisonFn() when a format or protocol explicitly restricts case-insensitive matching to ASCII letters.
Both methods return an empty comparison callback for case-sensitive mode. String APIs interpret an empty callback as exact Unicode code-point comparison.
auto mode = el::CaseSensitivity{el::CaseSensitivity::CaseInsensitive};
if (left.compare(right, mode.comparisonFn()) == std::strong_ordering::equal) {
// The strings match using Unicode simple case folding.
}
Char
Char stores one Unicode code point as a small value type.
Use it when you want to talk about a decoded character in Erbsland Core text APIs without mixing it up with a raw
char byte, a UTF-8 code unit, or a full grapheme cluster.
The type intentionally stays simple. It validates code point ranges, provides a few well-known values, and is cheap to copy. It does not model language-specific text behavior such as combining marks, collation, or grapheme boundaries.
Unicode Light Database
Some Char APIs use the Unicode Light database.
These are the category and case-mapping helpers such as
category(),
categoryGroup(),
caseFolded(),
toLowercase(), and
toUppercase().
If your code never calls one of those APIs, the generated Unicode Light database does not need to be linked into the final binary. That keeps small ASCII-only or encoding-only use cases lean while still making Unicode metadata available when you need it.
ASCII Fast Paths
For common ASCII checks, use isAscii() and the templated
isAscii() fast paths.
They do not use the Unicode Light database and are a good fit for protocol parsing, command-line tooling, and other text
formats that intentionally stay in the ASCII subset.
Use isAsciiWord() for the common ASCII letter, digit, or underscore set.
isSpecialRegexCharacter() identifies characters that must be
escaped when inserted literally into regular-expression syntax.
using Char = el::Char;
auto value = Char{U'F'};
if (value.isAsciiCategory(el::AsciiCategory::HexDigit)) {
// Fast ASCII-only path.
}
ASCII Categories
AsciiCategory names reusable ASCII-only character classes.
isAsciiCategory() tests them directly without constructing a set or
consulting the Unicode database.
The parser-oriented categories have these exact memberships:
Category |
Membership |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
ASCII letters and digits plus |
Base64Text deliberately includes = padding, but it does not validate padding placement.
HttpToken follows the HTTP token character definition and excludes separators, whitespace, controls, and non-ASCII
characters.
For sequential parsing, pass the category directly to the while/until operations of
StringCharReader. For whole-string validation, use the
containsOnly(AsciiCategory) overload on the read-only or editor string type.
Both paths classify decoded characters directly.
Malformed encoded input is decoded tolerantly to a replacement character, which never belongs to an ASCII category.
Integer Digit Helpers
digitValue() converts ASCII decimal and Latin letter digits into a
numeric value.
The overload accepting IntegerBase returns a value only if the character is a
valid digit for that base.
isDigitValue() combines that conversion with an
IntegerBase check.
Use fromDigitValue() when emitting integer digits with a selected
LetterCase.
Unicode Categories
Use category() when you need the exact Unicode general category and
categoryGroup() when the broad class is enough.
The convenience predicates
isCategory(),
isCategoryGroup(), and
is() help keep that intent visible in your code.
auto codePoint = el::Char{U'_'};
if (codePoint.isCategory(el::UnicodeCategory::ConnectorPunctuation)) {
// Handle connector punctuation explicitly.
}
Display Width
displayWidth() returns the approximate cell width for one Unicode code point.
It uses the Unicode Light database and returns 0 for Unicode control characters, invalid code points, and internal
signal values.
Use it for straightforward alignment and measuring tasks where a per-code-point width is enough.
Simple Case Mapping
caseFolded(),
toLowercase(), and
toUppercase() all use simple one-code-point mappings.
That makes them fast and predictable for character-wise processing.
This also means they intentionally do not perform full Unicode case mappings that can expand to multiple code points. For example, these helpers are meant for one code point in, one code point out.
Use toAsciiLowercase() and toAsciiUppercase() for ASCII-only folding.
These helpers only map A-Z and a-z and do not use the Unicode database.
Basic Usage
Create a Char from a char32_t code point and use
toRawValue() when you need the underlying value again.
auto letter = erbsland::text::Char{U'A'};
if (letter.isValidUnicode()) {
auto codePoint = letter.toRawValue();
}
The default value is the null character.
Use isNull() when this matters for your logic.
Signals
Some text APIs use reserved invalid Char values as non-character signals.
The reserved signal range is 0xFFFFFF00 through 0xFFFFFFFF.
Current signals are endOfData() with raw value 0xFFFFFFFF, noCodePoint() with raw value 0xFFFFFFFE, and
error() with raw value 0xFFFFFFFD.
The byteOrderMark() signal has raw value 0xFFFFFFFC and represents an encoding-independent BOM at a byte
encoding boundary.
Use isSignal(), isEndOfData(), isNoCodePoint(), isError(), and isByteOrderMark() when an internal
reader, writer, or character-processing algorithm can report these states.
The error signal is reserved for internal algorithms and is never returned by public Erbsland Core text APIs.
The BOM signal is similarly restricted to encoded-data boundaries.
Ordinary string readers, iterators, conversions, and content writers never expose or encode it.
Signals are not valid Unicode code points.
Validation
isValidUnicode() checks whether the value is a valid Unicode code
point.
It rejects values above U+10FFFF, surrogate code points, and U+FEFF.
Erbsland Core intentionally reserves U+FEFF for an initial encoded byte-order signature.
Consequently, Char{0xFEFF} is invalid text content and is distinct from the byteOrderMark() signal.
U+2060 WORD JOINER remains valid content.
When a decoder encounters invalid input in a tolerant API, it can return
replacement().
You can detect that value with isReplacement().
Use isSafeUnicode() before inserting a character directly into
diagnostics or other safe display text.
It rejects invalid values, controls, invisible formatting characters, and reserved display ranges without consulting the
Unicode Light database.
Char Range
CharRange describes a closed range of Unicode scalar values.
It is the range building block used by CharSet.
The type is tolerant at construction time. Invalid endpoints create an empty range, while two valid endpoints are ordered automatically. This keeps call sites simple when user input may provide the bounds in either order.
Unicode Scalar Values
The range stores only endpoint values. If a range spans the UTF-16 surrogate area, membership and exports still treat surrogate code points as invalid Unicode scalar values.
auto range = el::CharRange{el::Char{U'Z'}, el::Char{U'A'}};
if (range.contains(el::Char{U'M'})) {
// The endpoints were normalized to A-Z.
}
Char Set
CharSet stores a normalized set of Unicode scalar values.
It is the reusable character-set type for APIs such as
findFirstOf() and
containsOneOf().
The set stores up to two normalized ranges directly in the value without allocating memory.
Larger sets use one contiguous copy-on-write allocation, so passing and copying them remains cheap until a copy is
modified.
Adjacent and overlapping CharRange values are merged.
Duplicate characters and invalid code points are ignored.
Creating Sets
Create a set from individual characters, ranges, UTF-8 text, or standard-library containers. UTF-8 input is decoded tolerantly, matching the rest of the string API: invalid byte sequences contribute the replacement character.
using Char = el::Char;
using CharRange = el::CharRange;
using CharSet = el::CharSet;
auto digits = CharSet::fromRange(Char{U'0'}, Char{U'9'});
auto hexLetters = CharSet::fromRange(Char{U'A'}, Char{U'F'}) | CharSet::fromRange(Char{U'a'}, Char{U'f'});
auto separators = CharSet{u8",;"_el};
Use CharSet::fromRange() when two characters describe an inclusive
range.
The expression CharSet{Char{U'0'}, Char{U'9'}} creates a set containing only the two characters 0 and 9.
Use CharSet::from(AsciiCategory) only when an API requires a retained set or
the category must participate in set operations.
Direct category classification and scanning avoid materializing that set.
Use CharSet::from(UnicodeCategory) for sets derived from a Unicode general
category.
Use CharSet::fromPattern() for compact literal/range patterns where
- between two characters defines an inclusive range and a leading or trailing - is a literal hyphen.
auto optionName = el::CharSet::fromPattern("-a-zA-Z0-9_"_el);
Set Operations
Use named methods when clarity matters, or operators when the expression reads naturally.
auto identifierStart = letters | CharSet{Char{U'_'}};
auto identifierContinue = identifierStart.unitedWith(digits);
if (identifierStart <= identifierContinue) {
// Every start character is also allowed as a continuation character.
}
Case Helpers
toLowercase() and
toUppercase() apply simple one-code-point mappings and normalize
duplicates.
Use isEqualToIgnoringCase() and isSubsetOfIgnoringCase() for simple Unicode case-insensitive membership checks.
Exporting Characters
Use toList() or toSet() when another API needs the individual characters.
Use toString(), toU8String(), toU16String(), or toU32String() when the set should be materialized as
text in ascending code-point order.
Char Signal
CharSignal names reserved non-character values that can be stored in
Char.
Use these signals with reader and character-access APIs that must distinguish the end of available data from an invalid
or absent position.
EndOfData marks the exact position just after the last available character.
NoCodePoint marks an invalid, no-index, or out-of-range position.
Error is reserved for failures propagated inside character-processing algorithms; public Erbsland Core text APIs
never return it.
Unicode Category
UnicodeCategory represents one Unicode general category such as
uppercase letters, decimal numbers, or spacing separators.
You get it from Char::category() when you need the precise category for one
code point.
These values follow the standard Unicode general-category model. That makes them useful for validation, tokenization, and simple character classifiers where you want stable, well-known semantics instead of ad-hoc tables.
Unicode Category Group
UnicodeCategoryGroup is the coarse Unicode general-category group.
It lets you ask broader questions like “is this code point a letter?” or “is this code point a separator?” without
caring about the exact subcategory.
You usually reach it through Char::categoryGroup().
That keeps user code short and readable when the precise subcategory would add no extra value.
Combined Char
CombinedChar stores one base Unicode code point and up to two combining marks.
It is a small text-domain value type for places that need to keep a visually combined input character together without
modeling full grapheme-cluster rules.
The type normalizes unsupported input to the Unicode replacement character.
Use first() for the leading code point, singleOrNull() for fast single-code-point checks, and toString() or
toU32String() to materialize the stored sequence.
Interface
-
enum class erbsland::text::AsciiCategory : uint8_t
ASCII-only categories for fast classification without the Unicode database.
Values:
-
enumerator Letter
An ASCII letter:
[A-Za-z].
-
enumerator LowercaseLetter
An ASCII lowercase letter:
[a-z].
-
enumerator UppercaseLetter
An ASCII uppercase letter:
[A-Z].
-
enumerator Digit
An ASCII decimal digit:
[0-9].
-
enumerator HexDigit
An ASCII hexadecimal digit:
[0-9A-Fa-f].
-
enumerator Alphanumeric
An ASCII letter or decimal digit:
[A-Za-z0-9].
-
enumerator Word
An ASCII word character:
[_A-Za-z0-9].
-
enumerator WordWithHyphen
An ASCII word character or hyphen:
[-_A-Za-z0-9].
-
enumerator DottedName
An ASCII dotted-name character:
[-._A-Za-z0-9].
-
enumerator UrlScheme
An ASCII URL-scheme character:
[+\-.A-Za-z0-9].
-
enumerator Base64Text
A standard Base64 alphabet or padding character:
[+/=A-Za-z0-9].
-
enumerator HttpToken
An HTTP token character defined by RFC 9110.
-
enumerator Whitespace
An ASCII whitespace character: space or
[\t-\r].
-
enumerator Blank
An ASCII horizontal blank: space or tab.
-
enumerator Control
An ASCII control character:
[\x00-\x1f\x7f].
-
enumerator Punctuation
An ASCII punctuation character.
-
enumerator Letter
-
class CaseSensitivity
The case sensitivity used for character-wise text comparisons.
Public Types
Public Functions
-
inline constexpr CaseSensitivity(const Value value) noexcept
Create a case sensitivity from a value.
-
CharCompareFn comparisonFn() const noexcept
Get the character comparison callback using Unicode case folding.
For case-sensitive comparisons, this returns an empty callback to select exact code-point comparison.
-
CharCompareFn asciiComparisonFn() const noexcept
Get the character comparison callback using ASCII-only case folding.
For case-sensitive comparisons, this returns an empty callback to select exact code-point comparison.
-
inline constexpr CaseSensitivity(const Value value) noexcept
-
class Char
A single 32bit Unicode code-point.
Public Functions
-
inline constexpr bool operator==(const CharSignal signal) const noexcept
Test whether this character has a signal value.
-
inline constexpr bool operator!=(const CharSignal signal) const noexcept
Test whether this character does not have a signal value.
-
inline constexpr char32_t toRawValue() const noexcept
Access the raw underlying code-point.
-
inline constexpr bool isAsciiLowercaseLetter() const noexcept
Test if the Char is an ASCII lowercase letter.
-
inline constexpr bool isAsciiUppercaseLetter() const noexcept
Test if the Char is an ASCII uppercase letter.
-
inline constexpr bool isAsciiHexDigit() const noexcept
Test if the Char is an ASCII hexadecimal digit.
-
inline constexpr bool isAsciiAlphanumeric() const noexcept
Test if the Char is an ASCII alphanumeric character.
-
inline constexpr bool isAsciiWord() const noexcept
Test if the Char is an ASCII word character (letter, digit, or underscore).
-
inline constexpr std::optional<unsigned int> digitValue() const noexcept
Get the ASCII digit value.
-
inline constexpr std::optional<unsigned int> digitValue(const IntegerBase base) const noexcept
Get the ASCII digit value if it is valid for the given integer base.
-
inline constexpr bool isDigitValue(const IntegerBase base) const noexcept
Test if this character is a valid digit in the given integer base.
-
inline constexpr bool isAsciiWhitespace() const noexcept
Test if the Char is an ASCII whitespace character.
-
inline constexpr bool isAsciiBlank() const noexcept
Test if the Char is an ASCII blank character (space or tab).
-
inline constexpr bool isAsciiControl() const noexcept
Test if the Char is an ASCII control character.
-
inline constexpr bool isAsciiPunctuation() const noexcept
Test if the Char is an ASCII punctuation character.
-
inline constexpr bool isSpecialRegexCharacter() const noexcept
Test if the Char has a special meaning in regular-expression syntax.
-
inline constexpr bool isAsciiCategory(const AsciiCategory category) const noexcept
Test if the Char matches an ASCII-only category without using the Unicode Light database.
-
inline constexpr bool isNull() const noexcept
Test if the Char represents a null character (code-point 0).
-
inline constexpr bool isSignal() const noexcept
Test if the Char is a reserved non-character signal.
-
inline constexpr bool isNoCodePoint() const noexcept
Test if the Char represents the absence of a code point.
-
inline constexpr bool isError() const noexcept
Test if the Char represents an internal character-processing failure.
-
inline constexpr bool isByteOrderMark() const noexcept
Test if the Char represents the encoding-boundary byte-order-mark signal.
-
inline constexpr bool isValidUnicode() const noexcept
Test if the Char represents a valid Unicode code-point.
Erbsland Core reserves U+FEFF for encoded-data boundaries and rejects it as text content.
-
inline constexpr bool isSafeUnicode() const noexcept
Test if the Char can be displayed directly in diagnostics and other safe strings.
This rejects invalid values, controls, invisible formatting characters, and reserved display ranges.
-
UnicodeCategory category() const noexcept
Get the Unicode general category for this character.
-
inline UnicodeCategoryGroup categoryGroup() const noexcept
Get the Unicode general category group for this character.
-
inline bool isCategoryGroup(const UnicodeCategoryGroup expectedCategoryGroup) const noexcept
Test if the character is in the given Unicode general category group.
-
inline bool isCategory(const UnicodeCategory expectedCategory) const noexcept
Test if the character is in the given Unicode general category.
-
bool isControl() const noexcept
Test if the character is a Unicode control character.
-
bool isControlOrFormat() const noexcept
Test if the character is a Unicode control or format character.
-
int displayWidth() const noexcept
Get the approximate display width for this Unicode code point.
Invalid characters and Unicode control characters have width zero.
-
unit::ByteLength utf8Size() const noexcept
Get the size of the character in UTF-8 bytes.
-
unit::U16DataLength utf16Size() const noexcept
Get the size of the character in UTF-16 units.
-
std::size_t encodedSize(StringKind stringKind) const noexcept
The encoded size in the unit of the encoding.
For UTF-8, this is bytes (uint8_t). For UTF-16, this is words (uint16_t). For UTF-32, this is code points (uint32_t).
- Parameters:
stringKind – The kind of string encoding to query the size for.
- Returns:
The encoded size of the character in the specified string kind.
-
unit::ByteLength encodedBytes(StringEncoding encoding) const noexcept
Get the number of bytes required to encode this character.
Invalid Unicode values and character signals have an encoded length of zero.
- Parameters:
encoding – The target byte encoding.
-
Char caseFolded() const noexcept
Return the simple one-code-point case-folded form of this character.
-
inline constexpr Char toAsciiLowercase() const noexcept
Convert ASCII uppercase letters to lowercase without using the Unicode database.
-
inline constexpr Char toAsciiUppercase() const noexcept
Convert ASCII lowercase letters to uppercase without using the Unicode database.
-
inline constexpr Char toIdentifierNormalized() const noexcept
Convert to an identifier normalized character.
This is ASCII-only case-folded, and space (U+0020) is converted to an underscore (U+005F).
-
inline constexpr std::strong_ordering compareAsciiFolded(const Char other) const noexcept
Compare this character using ASCII-only case folding.
Public Static Functions
-
static inline constexpr bool isHighSurrogate(const char16_t value) noexcept
Test if the given UTF-16 code unit is a high surrogate.
-
static inline constexpr bool isLowSurrogate(const char16_t value) noexcept
Test if the given UTF-16 code unit is a low surrogate.
-
static inline Char caseFolded(Char character) noexcept
Return the simple one-code-point case-folded form of a character.
-
static inline Char toLowercase(Char character) noexcept
Convert a character to its simple lowercase form.
-
static inline Char toUppercase(Char character) noexcept
Convert a character to its simple uppercase form.
-
static inline constexpr Char toAsciiLowercase(const Char character) noexcept
Convert ASCII uppercase letters in a character to lowercase without using the Unicode database.
-
static inline constexpr Char toAsciiUppercase(const Char character) noexcept
Convert ASCII lowercase letters in a character to uppercase without using the Unicode database.
-
static inline constexpr Char toIdentifierNormalized(const Char character) noexcept
Convert to an identifier normalized character.
-
static std::strong_ordering compareAsciiFolded(Char left, Char right) noexcept
Compare two characters using ASCII-only case folding.
-
static std::strong_ordering compareCaseFolded(Char left, Char right) noexcept
Get a comparison callback that applies Unicode case folding.
-
static std::strong_ordering compareIdentifier(Char left, Char right) noexcept
Compare two characters using identifier folding.
-
static inline constexpr Char replacement() noexcept
Get the replacement character (U+FFFD).
This library uses the Unicode replacement character (U+FFFD) to represent invalid or unrepresentable characters.
-
static inline constexpr Char fromSignal(const CharSignal signal) noexcept
Create a Char from a reserved non-character signal.
-
static inline constexpr Char error() noexcept
Get the internal character-processing error signal.
This signal is reserved for internal algorithms and is never returned by public Erbsland Core text APIs.
-
static inline constexpr Char byteOrderMark() noexcept
Get the encoding-independent byte-order-mark signal.
This signal is reserved for encoded-data boundaries and is never exposed as text content.
-
static inline constexpr auto fromDigitValue(const unsigned int digit, const LetterCase letterCase = LetterCase::Lowercase) noexcept -> Char
Create an ASCII digit character from a value.
-
inline constexpr bool operator==(const CharSignal signal) const noexcept
-
using erbsland::text::CharCompareFn = std::strong_ordering (*)(Char left, Char right) noexcept
A function to compare two decoded characters.
-
class CharRange
A range of Unicode scalar values.
Empty ranges are represented by invalid endpoint characters. Invalid input creates an empty range, while valid endpoints are ordered automatically.
Public Functions
-
constexpr CharRange() noexcept = default
Create an empty range.
-
inline explicit constexpr CharRange(const Char character) noexcept
Create a range containing one character.
-
inline constexpr CharRange(const Char first, const Char second) noexcept
Create a range from two characters, ordering the endpoints automatically.
-
inline constexpr bool isEmpty() const noexcept
Test if this range is empty.
-
inline constexpr bool isSingleChar() const noexcept
Test if this range contains exactly one character.
-
inline constexpr bool contains(const Char character) const noexcept
Test if the character is contained in this range.
-
inline constexpr bool overlaps(const CharRange &other) const noexcept
Test if this range overlaps another range.
-
inline constexpr bool isAdjacentTo(const CharRange &other) const noexcept
Test if this range is directly adjacent to another range in Unicode scalar order.
-
inline constexpr bool canMergeWith(const CharRange &other) const noexcept
Test if this range can be merged with another range.
-
bool containsCaseFoldableCharacters() const noexcept
Test if this range contains characters affected by Unicode simple case folding.
-
bool containsLowercaseMappableCharacters() const noexcept
Test if this range contains characters affected by Unicode simple lowercase mapping.
-
bool containsUppercaseMappableCharacters() const noexcept
Test if this range contains characters affected by Unicode simple uppercase mapping.
-
constexpr CharRange() noexcept = default
-
class CharSet
A normalized set of Unicode scalar values.
The set stores up to two ranges inline and uses copy-on-write storage for larger sets. Invalid characters are ignored.
Public Functions
-
CharSet() = default
Create an empty character set.
-
explicit CharSet(const util::Set<Char> &characters)
Create a character set from an ordered Erbsland Core set of characters.
-
explicit CharSet(const util::List<Char> &characters)
Create a character set from an Erbsland Core list of characters.
-
inline bool operator!=(const CharSet &other) const noexcept
Test two character sets for inequality.
-
inline bool operator<=(const CharSet &other) const
Test whether this set is a subset of another set.
-
inline bool operator>=(const CharSet &other) const
Test whether this set is a superset of another set.
-
inline CharSet operator&(const CharSet &other) const
Return the intersection of two character sets.
-
inline CharSet operator^(const CharSet &other) const
Return the symmetric difference of two character sets.
-
inline bool isEmpty() const noexcept
Test if this set is empty.
-
inline bool isSupersetOf(const CharSet &other) const
Test if this set is a superset of another set.
-
bool isEqualToCI(const CharSet &other) const
Test if this set equals another set after Unicode simple case folding.
-
bool isSubsetOfCI(const CharSet &other) const
Test if this set is a subset of another set after Unicode simple case folding.
-
bool containsCaseFoldableCharacters() const
Test if this set contains characters affected by Unicode simple case folding.
-
bool containsLowercaseMappableCharacters() const
Test if this set contains characters affected by Unicode simple lowercase mapping.
-
bool containsUppercaseMappableCharacters() const
Test if this set contains characters affected by Unicode simple uppercase mapping.
-
CharSet intersectedWith(const CharSet &other) const
Create the intersection of this set and another set.
-
CharSet symmetricDifferenceWith(const CharSet &other) const
Create the symmetric difference of this set and another set.
-
template<typename Function>
util::LoopResult forEach(Function function) const Iterate over all ranges or characters in this set.
If the function accepts a
CharRange, ranges are iterated. Otherwise, if it accepts aChar, all Unicode scalar values are iterated in ascending order. If the function returnsLoopStatus,StoporErrorstops iteration.
Public Static Functions
-
static CharSet fromRange(Char from, Char to)
Create a character set containing one character range.
-
static CharSet from(AsciiCategory category)
Create a character set from an ASCII-only category.
-
static CharSet from(UnicodeCategory category)
Create a character set from a Unicode general category.
-
static CharSet from(UnicodeCategoryGroup categoryGroup)
This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
-
static CharSet fromPattern(const U8String &pattern)
Create a character set from a regexp like pattern.
The hypen character in the pattern defines ranges in the form
<first>-<last>. The code point of<first>must be before<last>. The hypen character at the beginning or end of the pattern is treated as a literal character. Duplicated characters and ranges are ignored. Example:fromPattern("-a-f_0-9=/")characters-_=/and rangesa-fand0-9.- Parameters:
pattern – The pattern string to parse.
- Throws:
err::ParseError – For an invalid pattern syntax.
-
CharSet() = default
-
enum class erbsland::text::CharSignal : uint8_t
Reserved non-character signals stored in
Charvalues.Values:
-
enumerator EndOfData
-
enumerator NoCodePoint
-
enumerator Error
An internal character-processing failure.
This signal is reserved for internal algorithms and is never returned by public Erbsland Core text APIs.
-
enumerator ByteOrderMark
An encoding-independent byte order mark for use at encoded-data boundaries.
-
enumerator EndOfData
-
class CombinedChar
Representation of one Unicode character with optional combining marks.
Public Types
Public Functions
-
constexpr CombinedChar() noexcept = default
Construct an empty character.
-
inline explicit constexpr CombinedChar(const Char codePoint) noexcept
Construct a character from a single code point.
- Parameters:
codePoint – The Unicode code point to store as the base character.
-
explicit CombinedChar(const String &text) noexcept
Construct a character from UTF-8 text.
- Parameters:
text – The UTF-8 text for exactly one terminal character. Invalid UTF-8 bytes are replaced per byte, and unsupported text normalizes to
U+FFFD. Empty input, control codes, and leading zero-width code points normalize toU+FFFD. Later visible code points also collapse the result toU+FFFD, while a third combining mark is ignored.
-
explicit CombinedChar(const U32String &text) noexcept
Construct a character from UTF-32 text.
- Parameters:
text – The UTF-32 text for exactly one terminal character. Invalid Unicode scalar values and unsupported text normalize to
U+FFFD. Empty input, control codes, and leading zero-width code points normalize toU+FFFD. Later visible code points also collapse the result toU+FFFD, while a third combining mark is ignored.
-
inline bool operator==(const CombinedChar &other) const noexcept
Compare two stored character sequences.
-
inline bool operator!=(const CombinedChar &other) const noexcept
Compare two stored character sequences.
-
inline constexpr Char singleOrNull() const noexcept
Get a single Unicode code point or zero for combined or empty characters.
This is a fast-path method for comparing a single-code point character, without the color.
- Returns:
The single code point, or
0if this character is combined or empty.
-
inline constexpr unit::CpLength characterCount() const noexcept
Get the number of stored code points.
-
inline int displayWidth() const noexcept
Get the display width of the leading character.
-
unit::ByteLength byteCount() const noexcept
Get the UTF-8 byte count for this sequence.
-
CombinedChar withCombining(Char codePoint) const noexcept
Create a copy with one additional combining code point.
- Parameters:
codePoint – The combining code point to append.
- Returns:
The updated character. Invalid combining code points and additions beyond the fixed storage are ignored.
-
inline constexpr bool isEmpty() const noexcept
Test if the character is empty.
-
inline bool isSpacing() const noexcept
Test if the character is spacing.
-
inline bool isControl() const noexcept
Test if the leading code point is a control code.
-
inline constexpr std::size_t hash() const noexcept
Get a stable hash for the stored code points.
Public Static Functions
-
static CombinedChar fromString(const String &text) noexcept
Parse exactly one visible text character from UTF-8 input.
Invalid or unsupported input normalizes to
U+FFFD.- Parameters:
text – The UTF-8 text to parse.
- Returns:
The parsed character.
-
static CombinedChar fromString(const U32String &text) noexcept
Parse exactly one visible text character from UTF-32 input.
Invalid or unsupported input normalizes to
U+FFFD.- Parameters:
text – The UTF-32 text to parse.
- Returns:
The parsed character.
-
constexpr CombinedChar() noexcept = default
-
using erbsland::text::ProcessCharacterFn = std::function<util::LoopStatus(Char character)>
A function that processes one decoded character.
Return
LoopStatus::Continueto continue iteration,LoopStatus::Stopto stop early, orLoopStatus::Errorto report an error.
-
using erbsland::text::ProcessCharacterWithCpIndexFn = std::function<util::LoopStatus(Char character, unit::CpIndex index)>
This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
-
using erbsland::text::TransformCharacterFn = Char (*)(Char character) noexcept
A function that maps one decoded character to a replacement character or signal.
Return
Char::endOfData()to stop transformation, orChar::noCodePoint()to skip the input character. The function can be called more than once for the same input while the implementation sizes the result. It must not rely on a particular call order or call count.
-
enum class erbsland::text::UnicodeCategory : uint8_t
A Unicode general category.
Values:
-
enumerator UppercaseLetter
An uppercase letter.
-
enumerator LowercaseLetter
A lowercase letter.
-
enumerator TitlecaseLetter
A titlecase letter.
-
enumerator ModifierLetter
A modifier letter.
-
enumerator OtherLetter
A letter of other kind.
-
enumerator NonspacingMark
A nonspacing combining mark.
-
enumerator SpacingMark
A spacing combining mark.
-
enumerator EnclosingMark
An enclosing combining mark.
-
enumerator DecimalNumber
A decimal digit.
-
enumerator LetterNumber
A letter-like numeric character.
-
enumerator OtherNumber
A numeric character of other kind.
-
enumerator ConnectorPunctuation
A connector punctuation mark.
-
enumerator DashPunctuation
A dash or hyphen punctuation mark.
-
enumerator OpenPunctuation
An opening punctuation mark.
-
enumerator ClosePunctuation
A closing punctuation mark.
-
enumerator InitialPunctuation
An initial quotation mark.
-
enumerator FinalPunctuation
A final quotation mark.
-
enumerator OtherPunctuation
A punctuation mark of other kind.
-
enumerator MathSymbol
A mathematical symbol.
-
enumerator CurrencySymbol
A currency symbol.
-
enumerator ModifierSymbol
A modifier symbol.
-
enumerator OtherSymbol
A symbol of other kind.
-
enumerator SpaceSeparator
A space separator.
-
enumerator LineSeparator
A line separator.
-
enumerator ParagraphSeparator
A paragraph separator.
-
enumerator Control
A control code.
-
enumerator Format
A format control character.
-
enumerator Surrogate
A surrogate code point.
-
enumerator PrivateUse
A private-use code point.
-
enumerator Unassigned
An unassigned or reserved code point.
-
enumerator UppercaseLetter
-
enum class erbsland::text::UnicodeCategoryGroup : uint8_t
The major Unicode general category group.
Values:
-
enumerator Letter
Letter (
L*) categories.
-
enumerator Mark
Mark (
M*) categories.
-
enumerator Number
Number (
N*) categories.
-
enumerator Punctuation
Punctuation (
P*) categories.
-
enumerator Symbol
Symbol (
S*) categories.
-
enumerator Separator
Separator (
Z*) categories.
-
enumerator Other
Other (
C*) categories.
-
enumerator Letter