Working with Character Sets

Many text-processing tasks start with a simple question:

“Which characters are allowed here?”

Whether you validate user input, parse configuration files, filter protocol fields, or normalize text, you often need a compact way to describe a set of accepted characters.

CharSet turns that question into a reusable value. It can describe a few explicit characters, broad Unicode categories, or a policy assembled from several smaller sets. The string APIs can then search, validate, trim, remove, or transform against the same definition.

This page begins with the Unicode character model behind those operations, then develops efficient ways to construct and reuse sets before applying them to realistic text policies. Along the way, it explains why encoding validation must happen before a character policy at an untrusted boundary and why a set of code points is not the same as a set of user-perceived characters.

Know What One Set Element Represents

Char represents one decoded Unicode code point. CharSet represents a set of those code points.

A character set does not represent grapheme clusters, locale-specific collation rules, or regular expressions. It answers questions such as:

  • Is this character a digit?

  • Is this character allowed in an identifier?

  • Does this string contain any control characters?

It does not answer questions that depend on multiple code points being interpreted together, such as a user-visible character composed from a base letter and one or more combining marks.

This distinction keeps character-set operations predictable and efficient. When your problem is about token syntax, configuration keys, command-line options, file-name filters, protocol fields, or simple cleanup passes, CharSet is usually the right tool.

Validate the Encoding Before the Character Policy

String operations that decode text are tolerant by default. For UTF-8 and UTF-16 input, malformed sequences are decoded as Char::replacement() in many inspection operations.

This behavior is useful for display, logging, and best-effort processing, but it is not the same as rejecting invalid input.

For security-sensitive or externally supplied input, validate the encoding before you apply character-level rules:

If tolerant processing is intentional, include Char::replacement() in the tested set or handle it explicitly. If malformed input must be rejected, validate the encoding before any tolerant operation can hide the distinction.

Build Expensive Policies Once and Reuse Them

Create character sets close to the policy they represent, but avoid rebuilding expensive sets in hot paths. CharSet uses copy-on-write storage, so passing instances around is inexpensive. Constructing large sets, especially those derived from Unicode categories, can be significantly more expensive than using them.

For validators, parsers, and repeated transformations, prefer static reusable sets:

/// Reuse named `CharSet` objects for validation policies that are applied repeatedly.
///
/// Building a set once makes the policy easier to read and avoids reconstructing category or pattern based sets in hot
/// paths.
void characterSetReusable() {
    static const auto optionNameChars = el::CharSet::fromPattern("-_a-zA-Z0-9"_el);

    const auto optionNames = el::StringList{
        "orbite-07"_el,
        "antenne_nord"_el,
        "équipe-science"_el,
        "module solaire"_el,
    };

    const auto yesNo = el::BooleanFormat::yesNo();
    optionNames.forEach([&](const el::String &optionName) -> void {
        el::io::printLine(optionName, " -> "_el, yesNo, optionName.containsOnly(optionNameChars));
    });
}

orbite-07 -> yes
antenne_nord -> yes
équipe-science -> no
module solaire -> no

Giving a set a descriptive name also makes the code easier to read. The validation rule becomes immediately visible without having to decode a pattern string.

Empty Sets

The default constructor creates an empty set:

/// The default `CharSet` constructor creates an empty set.
///
/// Empty sets are useful for disabled filters and for policies where no character is allowed.
void characterSetEmpty() {
    auto disabledFilter = el::CharSet{};
    const auto sample = el::String{"orbite"_el};

    const auto yesNo = el::BooleanFormat::yesNo();
    el::io::printLine("Set is empty ................: "_el, yesNo, disabledFilter.isEmpty());
    el::io::printLine("Sample contains one of set ..: "_el, yesNo, sample.containsOneOf(disabledFilter));
    el::io::printLine("Sample contains only set ....: "_el, yesNo, sample.containsOnly(disabledFilter));
    el::io::printLine("Empty text contains only set : "_el, yesNo, el::String{}.containsOnly(disabledFilter));
}

Set is empty ................: yes
Sample contains one of set ..: no
Sample contains only set ....: no
Empty text contains only set : yes

An empty set is useful when a configuration option disables filtering. It also provides a clear representation for policies where no characters are allowed.

For non-empty strings, containsOnly() returns false when the allowed set is empty. For an empty string, it returns true because no character violates the rule.

containsOneOf() always returns false for an empty set.

Create Sets from Characters and Ranges

Use direct construction for a small set of individual characters. Use CharSet::fromRange() when two characters describe an inclusive range and not two separate allowed characters. Char values make that intent explicit:

/// `CharSet` can be created directly from a single character or explicitly from an inclusive range.
///
/// Use `fromRange()` when two `Char` values describe bounds instead of two individual allowed characters.
void characterSetCharacters() {
    auto questionMark = el::CharSet{U'?'};
    auto asciiLowercase = el::CharSet::fromRange(U'a', U'z');

    const auto yesNo = el::BooleanFormat::yesNo();
    el::io::printLine("Question marker accepts '?' : "_el, yesNo, questionMark.contains(U'?'));
    el::io::printLine("Lowercase accepts 'm' ......: "_el, yesNo, asciiLowercase.contains(U'm'));
    el::io::printLine("Lowercase accepts 'M' ......: "_el, yesNo, asciiLowercase.contains(U'M'));
}

Question marker accepts '?' : yes
Lowercase accepts 'm' ......: yes
Lowercase accepts 'M' ......: no

The range factory creates an inclusive range. Ranges are normalized automatically, and adjacent or overlapping ranges are merged.

Create Sets from Text and Containers

When you already know the exact characters, construct the set from text or from a container:

/// `CharSet` can be created from text or from a list of decoded characters.
///
/// Duplicate characters are ignored, and the resulting set is normalized for efficient membership tests.
void characterSetText() {
    auto punctuation = el::CharSet{"!?.,;"_el};
    auto separators = el::CharSet{U',', U';', U':', U'/'};

    const auto message = el::String{"statut: prêt; orbite stable."_el};
    const auto yesNo = el::BooleanFormat::yesNo();
    el::io::printLine("Message contains punctuation : "_el, yesNo, message.containsOneOf(punctuation));
    el::io::printLine("Message contains separators .: "_el, yesNo, message.containsOneOf(separators));
    el::io::printLine("Separator accepts '/' .......: "_el, yesNo, separators.contains(U'/'));
}

Message contains punctuation : yes
Message contains separators .: yes
Separator accepts '/' .......: yes

Duplicate characters are ignored. Internally, the resulting set is stored as normalized ranges, so lookup performance is independent of the input order.

Create Sets from Patterns

Use CharSet::fromPattern() for compact literal and range patterns.

The syntax resembles the contents of a regular-expression character class, but it only describes characters and inclusive ranges.

/// `CharSet::fromPattern()` creates compact character sets from literal and range patterns.
///
/// A hyphen between two characters defines a range.
/// A leading or trailing hyphen is treated as a literal hyphen.
void characterSetPatterns() {
    auto identifierChars = el::CharSet::fromPattern("_a-zA-Z0-9"_el);
    auto optionNameChars = el::CharSet::fromPattern("-_a-zA-Z0-9"_el);

    const auto stationId = el::String{"ORBIT_07"_el};
    const auto optionName = el::String{"orbite-07"_el};
    const auto spacedName = el::String{"orbite 07"_el};

    const auto yesNo = el::BooleanFormat::yesNo();
    el::io::printLine("Identifier \"", stationId, "\" ....: "_el, yesNo, stationId.containsOnly(identifierChars));
    el::io::printLine("Option \"", optionName, "\" ........: "_el, yesNo, optionName.containsOnly(optionNameChars));
    el::io::printLine("Option \"", spacedName, "\" ........: "_el, yesNo, spacedName.containsOnly(optionNameChars));
}

Identifier "ORBIT_07" ....: yes
Option "orbite-07" ........: yes
Option "orbite 07" ........: no

A hyphen between two characters defines a range. A hyphen at the beginning or end of the pattern is treated as a literal hyphen.

Invalid ranges, such as z-a, throw ParseError.

Skip the Set When an ASCII Category Is Enough

An AsciiCategory is already a complete policy for common ASCII grammar classes such as digits, hexadecimal digits, words, dotted names, whitespace, and HTTP tokens. When an API accepts the category directly, constructing a CharSet first only allocates and builds a value that the operation does not need.

Use Char::isAsciiCategory() for one decoded character and String::containsOnly(AsciiCategory) for a complete string. StringCharReader also accepts categories directly in readWhile(), readUntil(), advanceWhile(), advanceUntil(), and the corresponding buffer operations. These overloads can test the compact category directly and may be faster than a set lookup.

Construct CharSet::from(AsciiCategory) when the set must be retained, combined with custom characters, or passed to an operation that only accepts CharSet.

Unicode categories require a set because they expand into ranges drawn from Unicode metadata. Use CharSet::from(UnicodeCategory) or CharSet::from(UnicodeCategoryGroup) when the policy should follow those standard character classes:

/// Use `AsciiCategory` directly when an API accepts it, and construct a `CharSet` only when the policy must be retained
/// or combined with other characters.
void characterSetCategories() {
    const auto packetId = el::String{"SENSOR-07"_el};
    const auto packetIdIsValid = packetId.containsOnly(el::AsciiCategory::WordWithHyphen);
    const auto firstIsUppercase =
        packetId.charAt(el::StringSide::Front).isAsciiCategory(el::AsciiCategory::UppercaseLetter);

    static const auto unicodeDigits = el::CharSet::from(el::UnicodeCategory::DecimalNumber);
    static const auto unicodeLetters = el::CharSet::from(el::UnicodeCategoryGroup::Letter);

    const auto yesNo = el::BooleanFormat::yesNo();
    el::io::printLine("Packet-id is ASCII word ..: "_el, yesNo, packetIdIsValid);
    el::io::printLine("First letter is uppercase : "_el, yesNo, firstIsUppercase);
    el::io::printLine("Unicode digit accepts '7' : "_el, yesNo, unicodeDigits.contains(U'7'));
    el::io::printLine("Unicode letter accepts 'é': "_el, yesNo, unicodeLetters.contains(U'é'));
}

Packet-id is ASCII word ..: yes
First letter is uppercase : yes
Unicode digit accepts '7' : yes
Unicode letter accepts 'é': yes

For testing individual characters, Char predicate methods are often the simplest solution.

Character sets become more useful when scanning strings repeatedly or when you need to combine categories with custom additions or exclusions.

Compose Larger Policies from Smaller Sets

CharSet supports the usual set operations.

Use the named methods when they make the policy easier to understand:

Use the operators |, &, -, and ^ when the resulting expression remains readable.

/// `CharSet` objects can be combined and compared to express larger validation policies.
///
/// Use set operations to build the final policy from named parts, then use subset checks when one policy must remain
/// inside another.
void combineCharacterSets() {
    static const auto letters = el::CharSet::from(el::UnicodeCategoryGroup::Letter);
    static const auto digits = el::CharSet::from(el::UnicodeCategory::DecimalNumber);
    static const auto identifierStart = letters | el::CharSet{U'_'};
    static const auto identifierContinue = identifierStart | digits | el::CharSet{U'-'};

    const auto identifier = el::String{"orbite-7"_el};
    const auto firstCharacterOk = identifierStart.contains(identifier.charAt(el::StringSide::Front));
    const auto fullIdentifierOk = identifier.containsOnly(identifierContinue);

    const auto yesNo = el::BooleanFormat::yesNo();
    el::io::printLine("Start policy is subset ....: "_el, yesNo, identifierStart.isSubsetOf(identifierContinue));
    el::io::printLine("First character accepted ..: "_el, yesNo, firstCharacterOk);
    el::io::printLine("Identifier accepted .......: "_el, yesNo, fullIdentifierOk);
}

Start policy is subset ....: yes
First character accepted ..: yes
Identifier accepted .......: yes

Set comparison is especially useful when part of a policy comes from user configuration. You can verify that an extension remains within a broader safe set before accepting it.

Ask Whether Text Satisfies a Character Policy

Use containsOnly() when every character must belong to the allowed set.

Use containsOneOf() when you only need to know whether a string contains at least one character from a set.

/// Combine encoding checks with `containsOnly()` and `containsOneOf()` for character-level validation.
///
/// Validate externally supplied text before applying tolerant decoded-character operations.
void validateCharacterPolicy() {
    static const auto userNameChars = el::CharSet::fromPattern("-_a-zA-Z0-9"_el);
    static const auto forbiddenChars = el::CharSet{" \t\r\n"_el};

    const auto userNames = el::StringList{
        "orbite-07"_el,
        "module solaire"_el,
        "équipe-science"_el,
    };

    const auto yesNo = el::BooleanFormat::yesNo();
    userNames.forEach([&](const el::String &userName) -> void {
        const auto isAccepted =
            userName.isValidUtf8() && userName.containsOnly(userNameChars) && !userName.containsOneOf(forbiddenChars);
        el::io::printLine(userName, " -> "_el, yesNo, isAccepted);
    });
}

orbite-07 -> yes
module solaire -> no
équipe-science -> no

The method names intentionally express the validation strategy.

containsOnly(allowed) describes an allow-list. containsOneOf(forbidden) describes a block-list or diagnostic check.

/// A validation error that can be thrown by the validation functions.
class ValidationError : public el::Exception {
public:
    explicit ValidationError(const el::String &message) : Exception(message) {}
};

/// Validate if the given email address is valid.
void validateEmailAddress(const el::String &emailAddress) {
    static const auto requiredAt = "@"_el;
    static const auto allowedDomainChars = el::CharSet::fromPattern("-a-zA-Z0-9."_el);
    static const auto allowedLocalChars = el::CharSet::fromPattern("-a-zA-Z0-9._+!#$%&'*=?^`{|}~"_el);
    if (!emailAddress.contains(requiredAt)) {
        throw ValidationError{"Email address must contain '@' character."_el};
    }
    if (emailAddress.count(requiredAt) > el::ItemCount{1}) {
        throw ValidationError{"Email address can only contain one '@' character."_el};
    }
    const auto indexOfAt = emailAddress.find(requiredAt);
    const auto domain = emailAddress.slice(el::ByteRange{indexOfAt + requiredAt.length(), el::ByteLength::infinite()});
    if (!domain.containsOnly(allowedDomainChars)) {
        throw ValidationError{"Email domain contains invalid characters."_el};
    }
    if (domain.isEmpty()) {
        throw ValidationError{"Email domain must not be empty."_el};
    }
    const auto local = emailAddress.slice(el::ByteRange{el::ByteIndex::zero(), indexOfAt});
    if (!local.containsOnly(allowedLocalChars)) {
        throw ValidationError{"Email local part contains invalid characters."_el};
    }
    if (local.isEmpty()) {
        throw ValidationError{"Email local part must not be empty."_el};
    }
}

/// This demo shows how text, character-set, and accepted-character tests can be used as an efficient input filter.
void testForCharacters() {
    auto emailAddressesToValidate = el::StringList{
        "tree🌲@forest.org"_el,
        "[email protected]"_el,
        "river@mountain!.org"_el,
        "[email protected]"_el,
        "space [email protected]"_el,
        "[email protected]"_el,
        "wolf@nächtlich.de"_el, // fails: unicode domain
        "[email protected]"_el,
        "missing-at-symbol.example.com"_el,
        "[email protected]"_el,
        "alice\n[email protected]"_el,
        "[email protected]"_el,
        "double@@example.com"_el, // fails: repeated at sign
        "forest@domain#name.com"_el,
        "[email protected]"_el,
        "@empty-local.org"_el,
        "[email protected]"_el,
        "user@exa mple.com"_el,
        "[email protected]"_el,
        "雨@example.jp"_el, // fails: unicode local part
        "[email protected]"_el,
        "invalid<char>@example.com"_el,
        "é[email protected]"_el, // fails: unicode local part
        "empty-domain@"_el,
        "[email protected]"_el,
    };

    el::io::printLine("Validating all "_el, emailAddressesToValidate.count(), " email addresses:"_el);
    emailAddressesToValidate.forEach([](const el::String &email) {
        el::io::print("- \"", email.toEscaped(el::EscapeFormat::Cpp), "\": "_el);
        try {
            validateEmailAddress(email);
            el::io::printLine("✅"_el);
        } catch (const ValidationError &error) {
            el::io::printLine("❌ "_el, error);
        }
    });
}

Validating all 25 email addresses:
- "tree🌲@forest.org": ❌ Email local part contains invalid characters.
- "anna.wald@example.com": ✅
- "river@mountain!.org": ❌ Email domain contains invalid characters.
- "maria.silva@green-energy.eu": ✅
- "space in@address.com": ❌ Email local part contains invalid characters.
- "takashi.yama@tokyo.jp": ✅
- "wolf@nächtlich.de": ❌ Email domain contains invalid characters.
- "luca+weather@forest-mail.net": ✅
- "missing-at-symbol.example.com": ❌ Email address must contain '@' character.
- "fatma+birds@forest.example": ✅
- "alice\nbob@example.com": ❌ Email local part contains invalid characters.
- "nora+rain@climate.example": ✅
- "double@@example.com": ❌ Email address can only contain one '@' character.
- "forest@domain#name.com": ❌ Email domain contains invalid characters.
- "carlos.sunrise@weather.es": ✅
- "@empty-local.org": ❌ Email local part must not be empty.
- "greta.wind@north-sea.dk": ✅
- "user@exa mple.com": ❌ Email domain contains invalid characters.
- "sofia.rivera@biology.org": ✅
- "雨@example.jp": ❌ Email local part contains invalid characters.
- "mehmet_istanbul@trees.dev": ✅
- "invalid<char>@example.com": ❌ Email local part contains invalid characters.
- "élise@fleurs.fr": ❌ Email local part contains invalid characters.
- "empty-domain@": ❌ Email domain must not be empty.
- "jan.kowalski@oakforest.pl": ✅

Trim a Policy from the Edges

Use trimmed() to remove selected characters from the front, the back, or both sides of a string.

Without arguments, the function trims ASCII whitespace from both ends. With a custom CharSet, it trims exactly the characters you specify.

/// `String::trimmed()` returns a view with selected characters removed from the front, back, or both sides.
///
/// With a custom `CharSet`, trimming is not limited to whitespace.
void trimCharacterSet() {
    const auto raw = el::String{"*** signal-orbite ;; "_el};
    static const auto border = el::CharSet{" *;"_el};

    auto clean = raw.trimmed(border);
    auto frontOnly = raw.trimmed(border, el::StringSide::Front);
    auto backOnly = raw.trimmed(border, el::StringSide::Back);

    el::io::printLine("Raw .......: \"", raw, "\""_el);
    el::io::printLine("Both sides : \"", clean, "\""_el);
    el::io::printLine("Front only : \"", frontOnly, "\""_el);
    el::io::printLine("Back only .: \"", backOnly, "\""_el);
}

Raw .......: "*** signal-orbite ;; "
Both sides : "signal-orbite"
Front only : "signal-orbite ;; "
Back only .: "*** signal-orbite"

For read-only strings, trimming returns a narrower owning slice into the original storage. No new string allocation is required unless you later materialize the result.

Remove Unwanted Characters Throughout the Text

Use removedAll() when unwanted characters may appear anywhere in the text:

/// `String::removedAll()` removes every decoded character from a selected `CharSet`.
///
/// This is useful for simple cleanup passes where unwanted characters may occur anywhere in the text.
void removeCharacters() {
    static const auto controlChars = el::CharSet::from(el::AsciiCategory::Control);

    const auto input = el::String{"rapport\torbite\nstable"_el};
    auto logLine = input.removedAll(controlChars);

    el::io::printLine("Original : "_el, input.toEscaped(el::EscapeFormat::Cpp));
    el::io::printLine("Cleaned  : "_el, logLine.toEscaped(el::EscapeFormat::Cpp));
}

Original : rapport\torbite\nstable
Cleaned  : rapportorbitestable

This is useful for log cleanup, user-facing diagnostics, and simple format-specific sanitizers.

The operation returns a string value. If no characters need to be removed, the implementation can reuse the original storage instead of allocating a modified copy.

Transform the Characters Selected by a Set

Use transformed() when every decoded character should pass through a mapping function.

For common transformations, Char already provides suitable function pointers:

/// `String::transformed()` can create a canonical text form with a single character-mapping function.
///
/// ASCII-only mappings are useful for machine-readable identifiers because they leave non-ASCII characters untouched
/// and avoid the Unicode database.
void canonicalStationName() {
    const auto displayName = el::String{"Module ORBITE-Äther 07"_el};
    auto canonicalName = displayName.transformed(el::Char::toAsciiLowercase);

    el::io::printLine("Display name ..: "_el, displayName);
    el::io::printLine("Canonical .....: "_el, canonicalName);
}

Display name ..: Module ORBITE-Äther 07
Canonical .....: module orbite-Äther 07

ASCII-only transformations are useful for machine-readable identifiers because they leave non-ASCII characters untouched and do not require Unicode case data.

Unicode-aware transformations use simple one-code-point mappings, which keeps the result length predictable.

You can also provide a custom mapping function. Return the original character to keep it, another character to replace it, or Char::noCodePoint() to remove it.

/// `String::transformed()` maps decoded characters into a new string.
/// You can use Unicode-aware operations, ASCII-only operations, or a custom mapping function.
/// The original storage can be reused when the transformation does not change the text.
void caseTransformation() {
    const auto title = el::String{"Forêt d'Été, Σκιερό Μονοπάτι"_el};

    // Normalize display text with Unicode-aware operations.
    el::io::printLine("Original ......: "_el, title);
    el::io::printLine("Lowercase .....: "_el, title.transformed(el::Char::toLowercase));
    el::io::printLine("Uppercase .....: "_el, title.transformed(el::Char::toUppercase));
    el::io::printLine("Case folded ...: "_el, title.transformed(el::Char::caseFolded));

    // Using only ASCII methods can be faster and can avoid linking the Unicode database into the executable.
    const auto sensorName = el::String{"TEMP-ÄSTHETIK-07"_el};
    el::io::printLine("\nSensor name ...: "_el, sensorName);
    el::io::printLine("ASCII lower ...:  "_el, sensorName.transformed(el::Char::toAsciiLowercase));

    // A custom transform can map individual decoded code points.
    const auto quietLabel = el::String{"wind: leise, regen: sanft"_el};
    const auto highlighted = quietLabel.transformed(
        [](const el::Char character) noexcept -> el::Char { return character == U':' ? U'→' : character; });
    el::io::printLine("\nCustom map ..:   "_el, highlighted);
}

Original ......: Forêt d'Été, Σκιερό Μονοπάτι
Lowercase .....: forêt d'été, σκιερό μονοπάτι
Uppercase .....: FORÊT D'ÉTÉ, ΣΚΙΕΡΌ ΜΟΝΟΠΆΤΙ
Case folded ...: forêt d'été, σκιερό μονοπάτι

Sensor name ...: TEMP-ÄSTHETIK-07
ASCII lower ...:  temp-Ästhetik-07

Custom map ..:   wind→ leise, regen→ sanft