C++ Tokenizer
el::cpp::CppTokenizer scans C++ source for lightweight source inspection.
It belongs to erbsland::text::cpp and is available through <erbsland/cpp/CppTokenizer.hpp>.
It shares immutable source storage and produces one token at a time.
Scanning
Construct a tokenizer from a String and call next() until isEnd() is true.
Alternatively, iterate CppTokenizer::tokens(source); the lazy generator omits the end token and retains the source
until iteration finishes.
Tokens retain their source slices independently of the tokenizer, generator, and original string object.
Copies of a tokenizer share source storage but maintain independent cursors.
using namespace el::text::literals;
for (const auto &token : el::cpp::CppTokenizer::tokens("class Example {};"_el)) {
if (token.kind() == el::cpp::CppToken::Kind::Identifier) {
inspectIdentifier(token.text(), token.characterPosition());
}
}
CppToken exposes its category, original text, and zero-based source positions through accessors.
characterPosition() returns the decoded code-point position for diagnostics.
position() returns the native UTF-8 byte index for slicing or matching tokens to source ranges without rescanning
preceding characters.
Both positions are relative to the input passed to the tokenizer, even when that input is itself a slice.
Whitespace is skipped.
Punctuation uses the longest supported spelling, so >>= is one token while ..x becomes ., ., and x.
Comments, literals, and preprocessing directives include their delimiters; logical-line tokens include the terminating
newline when present.
End tokens have empty text and report the source’s decoded length and byte length in the respective position accessors.
A default-constructed token is an end token without either position.
characterRange() and range() return the complete decoded and native source ranges.
The tokenizer records each range from the reader’s start and end cursors, so obtaining a token’s code-point length via
token.characterLength() requires no decoding pass.
length() returns the byte length; both length accessors read the stored range lengths in constant time.
Both ranges include the token delimiters but exclude whitespace skipped before the token.
End-token ranges are empty at end of input; default-token ranges are noRange().
Manually constructed tokens supply both ranges alongside the corresponding source slice.
Scope and Errors
The scanner recognizes identifiers including keywords, numeric tokens, punctuation, quoted and raw literals, comments,
and line-start preprocessing directives.
It retains backslash continuations inside directives and line comments.
It does not expand macros, select conditional branches, parse declarations, or validate C++ syntax.
Non-ASCII characters are accepted as identifier characters without checking the C++ identifier character tables.
Universal character names, digraphs, and general preprocessing line splicing are not interpreted.
Numeric tokens begin with an ASCII digit; for example, .5 is punctuation followed by a number.
Numeric values and suffixes are not validated, and a user-defined literal suffix is returned separately.
Malformed quoted or raw literals and unterminated block comments raise erbsland::err::ParseError with the decoded
code-point position at which scanning failed.
Generator errors occur during iteration.
Recovery after an error is not supported; construct a new tokenizer for another scan.
Interface
-
class CppToken
A lexical C++ token retaining shared ownership of its unmodified source slice.
See: C++ Tokenizer
Public Types
-
enum class Kind : uint8_t
The lexical categories recognized by the source scanner.
Values:
-
enumerator End
End of input; the text is empty.
-
enumerator Identifier
An identifier, including keywords.
-
enumerator Number
A numeric token, without validating its value or suffix.
-
enumerator Punctuation
An operator, delimiter, or otherwise unclassified character.
-
enumerator Literal
A quoted character, string, or raw string literal.
-
enumerator Comment
A line or block comment, including its delimiters.
-
enumerator Directive
A complete preprocessing directive, including line continuations.
-
enumerator End
Public Functions
-
CppToken() = default
Create an end token without a source position.
-
inline CppToken(Kind kind, String text, unit::ByteRange range, unit::CpRange characterRange)
Create a token from its category and source slice.
- Parameters:
kind – The lexical category.
text – The unmodified source slice, including delimiters.
range – The UTF-8 byte range corresponding to the source slice.
characterRange – The decoded code-point range corresponding to the same source slice.
-
inline bool isEnd() const noexcept
Test whether this token marks the end of input.
-
inline const String &text() const noexcept
Get the source slice; its storage remains valid independently of the tokenizer.
-
inline unit::ByteIndex position() const noexcept
Get the UTF-8 byte offset, or no-index for a default-constructed token.
-
inline unit::ByteRange range() const noexcept
Get the UTF-8 byte range of the original source slice.
A tokenizer end token has an empty range at end of input; a default token has no-range.
-
inline unit::CpCount characterLength() const noexcept
Get the precomputed decoded code-point length in constant time.
-
enum class Kind : uint8_t
-
class CppTokenizer
Scan C++ source into tokens while sharing immutable source storage.
See: C++ Tokenizer
Public Functions
-
explicit CppTokenizer(const String &source)
Bind a source string and begin scanning at its first code point.
- Parameters:
source – The immutable source; its storage is retained by the tokenizer.
-
CppToken next()
Skip whitespace and read the next token, retaining comments, directives, and literals.
- Throws:
err::ParseError – For an unterminated comment or literal, or an invalid raw-string delimiter.
- Returns:
The next token, or an End token at the source length on every call after the last token.
Public Static Functions
-
static util::CoGenerator<CppToken> tokens(String source)
Iterate tokens while retaining the source in the coroutine; omit the End token.
- Parameters:
source – The source string whose storage the coroutine retains.
- Throws:
err::ParseError – During iteration if a comment or literal is malformed.
- Returns:
A lazy sequence of tokens in source order.
-
explicit CppTokenizer(const String &source)