Text Input, Output, and Encoding
Operating systems and file formats store text as bytes, while application code should work with Unicode characters. Text streams provide the boundary between these two views. They keep encoding details, incomplete byte sequences, and byte-order marks out of parsers and report writers, so those components can concentrate on the meaning of the text.
This boundary also preserves the safety rules of the stream API. Input is returned only at decoded code-point boundaries, allocating operations have explicit limits, and every read or write has a bounded wait. Timeout remains an ordinary result that you can retry or turn into an application-level cancellation; malformed text and failed I/O remain distinct errors.
This page explains how to choose a text operation, process blocks and lines without losing data, produce atomic output, capture output in memory, and select an encoding and recovery policy. The first step is to understand how the text interfaces fit together.
Choose the Text Operation That Matches the Task
TextInputStream adds Unicode decoding to the common
InputStream contract.
TextOutputStream performs the opposite transformation: callers write
characters and strings, and the stream encodes complete requests for its byte destination.
AnyStringBuilderStream implements the output interface without a byte
destination and collects the written text in memory.
Choose the input operation from the structure you expect, not merely from what is convenient at the call site:
readChar()Reads one decoded code point. Use it for tokenizers and parsers in which individual characters have meaning.
read(maximum)Returns one available block with at most the requested number of code points. Use it for incremental transforms, search, display, and other work that does not require line boundaries.
readLine(maximum)Collects one line, a maximum-length line fragment, or the final unterminated line. Use it for line-oriented formats while keeping a strict memory bound.
readAll(maximum)Collects text until normal end or the limit is reached. Reserve it for sources that are expected and required to remain small.
All four operations return a StreamReadResult.
Data makes the payload available, Finished reports normal end without another payload, and Timeout returns
control because no complete result became available before the configured deadline.
As with Byte Input and Output, only access data() after hasData() is true.
Failures of the stream or backing source throw StreamError instead of
masquerading as a normal result.
The distinction is deliberate. A timeout often means “try again later.” Normal end means the document is complete. A stream error means the source can no longer satisfy the operation. Keeping these outcomes separate lets a caller choose an appropriate retry, completion, or diagnostic path.
Process Unicode Without Splitting Encoded Characters
readChar() returns one Unicode code point, while
read() returns up to a finite number of code points.
The decoder may need several source bytes to produce one code point, so it retains an incomplete UTF-8, UTF-16, or
UTF-32 sequence internally until it can return a complete character.
Your processing loop never has to join partial encoded sequences.
The maximum is a CpLength rather than a byte count because the caller is already
on the Unicode side of the boundary.
A block may contain fewer code points than requested when that is all the source can currently provide.
It never ends in the middle of one encoded code point, although it can still end between user-perceived grapheme
clusters made from several code points.
Use a modest block size when processing a large or open-ended source.
This bounds temporary memory and lets the application respond between operations.
Use readAll() only after choosing a limit that is safe for the
data format; its no-argument overload still applies a library default, but an application-specific limit communicates
the expected document size more clearly.
An aggregate readAll() retains text collected before a timeout.
Repeat the same operation with the same maximum to continue it.
Plain read() is different: every Data result is already a complete block for the caller to process immediately.
In either case, combine the per-operation timeout configured when the stream was opened with an application-level
attempt or elapsed-time limit so a permanently stalled source cannot keep the workflow alive forever.
The following demo receives a generic text input stream; its hidden setup only creates the sample file and configures a one-second operation timeout. The visible function asks for at most eight code points, processes every available block, and resets its consecutive timeout count whenever progress is made. After three consecutive timeouts it stops the workflow, and a failed source is wrapped with the context that plant notes were being read.
/// Read decoded text in bounded, code-point-safe blocks.
/// Handle normal end, timeout, and stream failure separately; a finite retry limit prevents a stalled source from
/// keeping the application in this loop forever.
void readUnicodeBlocks(el::TextInputStream &input) {
constexpr auto cBlockLength = el::CpLength{8U};
constexpr auto cMaximumTimeouts = 3U;
auto consecutiveTimeouts = 0U;
try {
while (true) {
const auto result = input.read(cBlockLength);
if (result.isFinished()) {
return;
}
if (result.isTimeout()) {
if (++consecutiveTimeouts == cMaximumTimeouts) {
throw el::RuntimeError{"Too many timeouts occurred while reading the plant notes."_el};
}
continue;
}
consecutiveTimeouts = 0U;
el::io::printLine("Block: ["_el, result.data(), "]"_el);
}
} catch (const el::StreamError &) {
throw el::RuntimeError{"The plant notes could not be read."_el, std::current_exception()};
}
}
Block: [Frö 🌱 – ]
Block: [två blad]
Block: [ – höjd ]
Block: [7 cm]
Read Lines Without Losing Their Structure
readLine() treats a decoded line-feed character as the end of
a line and includes that character in the returned text.
For CRLF input, the preceding carriage return is preserved as part of the line as well.
Keeping the delimiter lets a formatter, proxy, or editor reproduce the original line structure instead of guessing which
returned lines were terminated.
The final line of a file does not need a delimiter.
If text remains when the source ends, readLine() first returns that unterminated text as Data; only the next
call returns Finished.
An empty Finished result therefore means there is no more line content, not that the preceding line should be
discarded.
Always pass a finite maximum based on the format you accept.
If no line feed appears within that maximum, the function returns a maximum-length fragment as Data.
The caller can process fragments as a streaming line, reject the record as too long, or accumulate it under a separate
application limit.
This design prevents a malformed or hostile input containing one endless line from forcing unbounded allocation.
If a timeout occurs while a line is incomplete, the stream retains the decoded prefix.
Repeat readLine() with the same maximum to continue that logical line; do not request only the missing suffix.
Selecting another read operation or changing the maximum cancels the aggregate line operation, but the retained text is
replayed to the newly selected operation in order.
The demo reads a small sequence of plant measurements.
It prints the line ending already present in each complete line and adds one only for the final unterminated line.
The three-attempt policy bounds a stalled input, while Finished and StreamError take separate completion and
failure paths.
/// Read lines with a finite length and a finite retry policy.
/// `readLine()` keeps a line ending when one is present and returns a final unterminated line as data.
/// Repeating the same call after timeout continues the pending logical line.
void readObservationLines(el::TextInputStream &input) {
constexpr auto cMaximumLineLength = el::CpLength{40U};
constexpr auto cMaximumTimeouts = 3U;
auto lineNumber = 1;
auto consecutiveTimeouts = 0U;
try {
while (true) {
const auto result = input.readLine(cMaximumLineLength);
if (result.isFinished()) {
return;
}
if (result.isTimeout()) {
if (++consecutiveTimeouts == cMaximumTimeouts) {
throw el::RuntimeError{"Too many timeouts occurred while reading the measurement series."_el};
}
continue;
}
consecutiveTimeouts = 0U;
el::io::print("Line "_el, lineNumber++, ": "_el, result.data());
if (!result.data().endsWith("\n"_el)) {
el::io::writeLine();
}
}
} catch (const el::StreamError &) {
throw el::RuntimeError{"The measurement series could not be read."_el, std::current_exception()};
}
}
Line 1: Dag 1: 2 cm
Line 2: Dag 8: 7 cm
Line 3: Dag 15: 13 cm
Produce Text as Complete Atomic Requests
Use write() when a character or string is already available.
Use writeLine() to append one line-feed character to an
existing string as part of the same request.
These low-level methods avoid building another formatting context and are the right choice for large, already-built
text.
Note
Text output assumes that Erbsland Core strings already contain the representation the application intends to write.
When the target uses the same representation, the stream copies it without another validation pass.
When transcoding is required, malformed source sequences become Unicode replacement characters.
Call isValid() before writing when the application must reject malformed internal text.
print() and
printLine() combine strings, characters, numbers, booleans,
and formatting objects concisely.
The stream first builds the formatted text and then commits it as one request, which is useful for a small record that
must not be interleaved with output from another caller.
For large generated content, prefer several bounded write() calls so the temporary formatted value does not grow
without a limit.
Every public text write is atomic with respect to timeout.
Success means the complete encoded request was accepted; Timeout means none of that request was accepted.
After timeout, retry the same unchanged call.
Do not resume at an estimated character or byte offset because there is no partial caller-visible acceptance to account
for.
Atomicity does not make an unlimited retry loop safe.
Give each stream a suitable operation timeout, then stop, reschedule, or cancel after a finite application-level budget.
A broken destination throws StreamError and should normally be caught where you can add useful context.
The component that opened an output stream also remains responsible for a successful
close(); a helper that merely receives TextOutputStream & should
not close a destination it does not own.
The report demo hides its in-memory destination and documents only a destination-independent writer.
Its small retry helper repeats each whole operation at most three times.
It uses write... for text that is already built, printLine() for the mixed text and numeric record, and wraps
native stream failure with report-specific context.
/// Write a report with a bounded retry policy for every atomic text operation.
/// Retry the complete unchanged call after timeout because the stream accepted none of it.
/// Convert a stream failure into an application error where useful report context is available.
void writeGrowthReport(el::TextOutputStream &output) {
constexpr auto cMaximumAttempts = 3U;
const auto writeWithRetry = [&](const auto &operation) -> void {
for (auto attempt = 0U; attempt < cMaximumAttempts; ++attempt) {
if (operation().isSuccess()) {
return;
}
}
throw el::RuntimeError{"The growth report timed out."_el};
};
try {
writeWithRetry([&]() { return output.writeLine("Tillväxtrapport"_el); });
writeWithRetry([&]() { return output.write("Mätserie: "_el); });
writeWithRetry([&]() { return output.writeLine("björkplantor"_el); });
writeWithRetry([&]() { return output.printLine("Dag "_el, 21, ": "_el, 18, " cm 🌿"_el); });
} catch (const el::StreamError &) {
throw el::RuntimeError{"The growth report could not be written."_el, std::current_exception()};
}
}
Tillväxtrapport
Mätserie: björkplantor
Dag 21: 18 cm 🌿
Capture Reusable Text Output in Memory
AnyStringBuilderStream is useful when an existing component writes to TextOutputStream but the caller needs the
result as a string.
Typical uses include report generation, templates, unit tests, and scoped capture of standard output.
The producer stays independent of whether its eventual destination is a file, terminal, redirected stream, or memory.
Unlike a file or pipe, the builder performs no external I/O. It is always ready, and its write operations do not time out. That makes it unnecessary to add a retry loop merely for the builder, but a reusable producer should still return or propagate the write status so another destination can apply its own policy.
Use AnyStringBuilderStream::toString() when the builder
must retain its content and the caller needs a copy.
Use AnyStringBuilderStream::takeString() when output
is complete and ownership should move out efficiently.
Taking the string resets the builder, so it can be reused for another capture without retaining the previous content.
The demo separates production from capture.
writePlantSummary() emits the whole two-line summary as one atomic request and returns its status.
buildPlantSummary() supplies a AnyStringBuilderStream, verifies the invariant that this local destination cannot
time out, and moves the completed string to its caller.
/// Produce a complete summary as one atomic text-stream request.
/// Returning the status lets a file or terminal caller apply its own retry policy without coupling this producer to
/// a particular destination.
auto writePlantSummary(el::TextOutputStream &output) -> el::StreamWriteStatus {
return output.print("Art: fjällsippa\nNya blad: "_el, 5, "\n"_el);
}
/// Capture text-stream output in memory and move the completed string out of the builder.
/// `AnyStringBuilderStream` is always ready and performs no external I/O, so this operation needs no timeout retry.
/// The same stream-oriented writer can also target a file or terminal.
auto buildPlantSummary() -> el::String {
const auto builder = el::AnyStringBuilderStream::create();
if (writePlantSummary(*builder).isTimeout()) {
throw el::LogicError{"An in-memory string builder unexpectedly timed out."};
}
return builder->takeString();
}
Captured report:
Match the Encoding and BOM Policy to the Format
Encoding is a property of the boundary, not of the application text.
Choose it when the stream is opened through
PathReadTextOptions or
PathWriteTextOptions.
After opening, readers and writers continue to use the same Erbsland Core string and character types regardless of the byte
representation below them.
StringEncoding supports UTF-8 plus UTF-16 and UTF-32 in generic or explicit
byte order.
Use UTF-8 for ordinary new formats unless an external specification requires something else.
Use generic Utf16 or Utf32 when a BOM may determine the byte order at the start of a document.
Use an explicit little- or big-endian variant when the format defines that order, or when positioning must begin away
from byte zero and no BOM can be inspected there.
StringBomMode makes the format contract explicit:
AutomaticAccepts a leading BOM on input. Generic UTF-16 and UTF-32 use its byte order and otherwise fall back to little-endian; output adds a BOM for every UTF-16 and UTF-32 variant but not for UTF-8.
RequireRejects input without a BOM and always writes one on output. Use it when the surrounding format relies on a signature and byte-order declaration.
RejectRejects a BOM on input and never writes one on output. Use it when a protocol or embedded text field specifies the encoding out of band.
These policies apply only at the initial byte boundary.
Readers consume an accepted signature before producing text, and writers emit it through their dedicated BOM path.
A repeated or embedded encoded U+FEFF is invalid input content and follows the input stream’s encoding mode; no
public string, reader, or iterator exposes an internal BOM signal.
encoding() reports the configured value.
effectiveEncoding() reports the concrete encoding
after the byte order has been resolved.
For example, an output configured as generic Utf16 reports Utf16LittleEndian as its effective encoding.
On input, code that needs the effective order should query it after enough leading data has been consumed to resolve a
possible BOM.
The demo opens a UTF-16 file with a required BOM and a one-second operation timeout. It retries the complete line at most three times, then checks both configured and effective encodings. Finally, it closes the owned file explicitly; a close timeout aborts the sample rather than silently abandoning queued output, and a native stream failure is reported with encoding-test context.
/// Open a bounded UTF-16 text stream and require a byte-order mark.
/// Retry the unchanged atomic write after timeout, verify the resolved byte order, and close the stream explicitly.
void writeEncodedMeasurement(const el::Path &path) {
constexpr auto cMaximumAttempts = 3U;
auto options = el::PathWriteTextOptions{el::StringEncoding::Utf16};
options.setBomMode(el::StringBomMode::Require);
options.setTimeout(el::TimeDelta::seconds(1));
try {
const auto output = path.content().openTextOutputStream(options);
for (auto attempt = 0U; attempt < cMaximumAttempts; ++attempt) {
if (output->writeLine("Rötter: 12 cm"_el).isSuccess()) {
el::io::printLine("Configured UTF-16: "_el, output->encoding() == el::StringEncoding::Utf16);
el::io::printLine("Effective little-endian: "_el,
output->effectiveEncoding() == el::StringEncoding::Utf16LittleEndian);
if (output->close().isTimeout()) {
output->abort();
throw el::RuntimeError{"Closing the encoding sample timed out."_el};
}
return;
}
}
output->abort();
throw el::RuntimeError{"The encoding sample timed out."_el};
} catch (const el::PathError &) {
throw el::RuntimeError{"The encoding sample could not be opened."_el, std::current_exception()};
} catch (const el::StreamError &) {
throw el::RuntimeError{"The encoding sample could not be written."_el, std::current_exception()};
}
}
Configured UTF-16: true
Effective little-endian: true
Choose Deliberately How Malformed Text Is Handled
Malformed encoded input is different from failed I/O.
The source may be readable and complete while its bytes are not valid for the selected encoding.
EncodingMode lets the boundary decide whether an application may
continue with a lossy interpretation:
TolerantInserts the Unicode replacement character for malformed input. This is a useful default for display, diagnostics, and imported prose where preserving progress matters most.
StrictRaises
EncodingErrorat the first malformed sequence. Use it for configuration, source code, identifiers, signed text, and other formats where replacement could change meaning or conceal corruption.
Set the mode before opening the stream.
Choosing Strict does not turn a timeout into an exception: bounded flow control still returns Timeout.
Likewise, an inaccessible or failed source still throws StreamError.
Catch EncodingError when you can reject or report invalid content, and catch StreamError where you can explain
which source operation failed.
The final demo creates malformed UTF-8 in hidden setup and passes only its path to the documented function.
The visible code selects strict decoding, limits both the aggregate read size and the wait for each attempt, and repeats
the unchanged readAll() after timeout.
It treats the expected EncodingError, a stalled source, unexpectedly accepted input, and failed I/O as four distinct
outcomes.
/// Reject malformed UTF-8 when silently repairing the input could change its meaning.
/// Bound each aggregate read, repeat it after timeout, and distinguish decoding errors from failed I/O.
void rejectInvalidEncoding(const el::Path &path) {
constexpr auto cMaximumAttempts = 3U;
auto options = el::PathReadTextOptions{el::StringEncoding::Utf8};
options.setEncodingMode(el::EncodingMode::Strict);
options.setTimeout(el::TimeDelta::seconds(1));
try {
const auto input = path.content().openTextInputStream(options);
for (auto attempt = 0U; attempt < cMaximumAttempts; ++attempt) {
if (!input->readAll(el::CpLength{100U}).isTimeout()) {
throw el::RuntimeError{"The damaged text was unexpectedly accepted."_el};
}
}
throw el::RuntimeError{"The damaged text timed out."_el};
} catch (const el::EncodingError &) {
el::io::printLine("Invalid UTF-8 was rejected."_el);
} catch (const el::PathError &) {
throw el::RuntimeError{"The encoded text could not be opened."_el, std::current_exception()};
} catch (const el::StreamError &) {
throw el::RuntimeError{"The encoded text could not be read."_el, std::current_exception()};
}
}
Invalid UTF-8 was rejected.