Reading Base-N Text into Bytes

An encoded field may come from a file, a network message, or another application’s output. Before using its bytes, you need to know whether the text follows the expected format and whether the decoded value fits the field’s size limit.

BaseNDecoder performs those checks while recovering a ByteBlock. This page explains the two failure-handling methods, the accepted whitespace and padding policies, and what strict decoding rejects. For the basic idea behind Base-N, start with Encoding Byte Data as Base-N Text.

Decode with a Byte Limit

Construct a decoder from the text field. It accepts the Core string widths and uses padded Base64 by default. When the protocol specifies another alphabet or layout, pass the same BaseNFormat that describes that field. The decoder’s format() accessor lets you inspect the selected format.

Both decoding methods accept a maximum decoded byte length. Set it at an input boundary when the field has a known limit. The limit applies to the resulting bytes, rather than to the number of characters in the encoded text.

Here a five-byte value fits a five-byte limit. The same text fails with a four-byte limit, while a separate malformed value fails because it is not valid Base64.

text/BaseN/BaseNDemos.cpp

void decodeSoundTag() {
    // Decode a small value with the largest byte count this field permits.
    const auto decoded = el::base_n::BaseNDecoder{"a2Fpa3U="_el}.toData(el::ByteCount{5U});
    el::io::printLine("Decoded bytes: "_el, decoded->length().toSizeT());

    // A malformed payload and an oversized payload both fail in the optional form.
    const auto invalid = el::base_n::BaseNDecoder{"a2Fpa3U!"_el}.toData();
    const auto tooLarge = el::base_n::BaseNDecoder{"a2Fpa3U="_el}.toData(el::ByteCount{4U});
    el::io::printLine("Malformed accepted: "_el, invalid.has_value());
    el::io::printLine("Oversized accepted: "_el, tooLarge.has_value());
}

Output

Decoded bytes: 5
Malformed accepted: false
Oversized accepted: false

toData() returns an optional byte block. An empty optional means either the text was malformed or the decoded size was too large. This is useful when a parser only needs to accept or reject a field.

toDataOrThrow() keeps the reason available. Malformed text raises ParseError. An exceeded byte limit raises OutOfRangeError. Use this form when diagnostics or different recovery paths depend on the reason.

Decide Which Characters to Ignore

The format’s whitespace() set tells the decoder which characters it may skip between encoded symbols. The predefined formats accept ASCII whitespace. That is why a wrapped Base64 payload can be decoded with its matching format.

setWhitespace() replaces the set. The example accepts a tilde between digits and then shows that an ordinary space is no longer ignored.

text/BaseN/BaseNDemos.cpp

void chooseWhitespace() {
    auto format = el::base_n::BaseNFormat::base64();
    format.setWhitespace(el::CharSet{U'~'});
    el::io::printLine("Tilde accepted: "_el, el::base_n::BaseNDecoder{"a2Fp~a3U="_el, format}.toData().has_value());
    el::io::printLine("Space accepted: "_el, el::base_n::BaseNDecoder{"a2Fp a3U="_el, format}.toData().has_value());
}

Output

Tilde accepted: true
Space accepted: false

The ignored set must not overlap the alphabet or the padding character. If the same character had two meanings, the decoder could not reliably decide whether it was data or a separator. The format validates this rule when you change the set.

Choose a Padding Policy

Standard Base32 and Base64 formats require canonical padding when the final group needs it. For example, the Base64 text for kaiku ends with one =. The format’s padding() character tells the decoder what to recognize, and the RequirePadding flag says whether it must be present.

Some protocols allow the padding to be omitted. For such a field, clear RequirePadding on the matching format. The next example shows the difference from the default policy.

text/BaseN/BaseNDemos.cpp

void decodePaddingPolicy() {
    auto format = el::base_n::BaseNFormat::base64();
    el::io::printLine(
        "Default accepts a2Fpa3U: "_el, el::base_n::BaseNDecoder{"a2Fpa3U"_el, format}.toData().has_value());

    format.clearFlags(el::base_n::BaseNFormatFlag::RequirePadding);
    el::io::printLine(
        "Optional padding accepts a2Fpa3U: "_el, el::base_n::BaseNDecoder{"a2Fpa3U"_el, format}.toData().has_value());
    el::io::printLine(
        "Incorrect padding accepted: "_el, el::base_n::BaseNDecoder{"a2Fpa3U=="_el, format}.toData().has_value());
}

Output

Default accepts a2Fpa3U: false
Optional padding accepts a2Fpa3U: true
Incorrect padding accepted: false

Optional padding does not mean arbitrary padding. If padding is present, the decoder still requires the canonical number of characters in the correct position. The encoder’s EmitPadding flag is a separate setting; it controls what is written, as explained in Writing Bytes as Base-N Text.

Strict Base-N Validation

After removing the configured ignored characters, the decoder examines the complete encoded value. It rejects characters outside the selected alphabet and alphabet characters that appear after padding. It also rejects missing required padding, non-canonical padding, and impossible final group lengths.

The last encoded character may contain a few bit positions that are not used by the recovered bytes. Those unused positions must be zero. Rejecting nonzero trailing bits avoids accepting multiple textual spellings for the same bytes. That matters when an encoded value is stored, signed, or compared with another value later.

The format must match the source. Ordinary Base64 and Base64url have different alphabets, even though many short encoded values happen to look the same. For wrapped output, the line separator must be part of the format’s ignored whitespace set. When you control both sides, use the same format object or the same factory and settings for writing and reading.

Complete demo sources

text/BaseN/BaseNDemos.cpp

// Copyright (c) 2026 Tobias Erbsland - https://erbsland.dev
// SPDX-License-Identifier: Apache-2.0

#include <DemoCommon.hpp>
#include <erbsland/base_n/BaseNDecoder.hpp>
#include <erbsland/base_n/BaseNEncoder.hpp>
#include <erbsland/base_n/BaseNFormat.hpp>
#include <erbsland/base_n/BaseNFormatFlag.hpp>
#include <erbsland/Byte.hpp>
#include <erbsland/ByteBlock.hpp>
#include <erbsland/ByteCount.hpp>
#include <erbsland/Char.hpp>
#include <erbsland/CharSet.hpp>
#include <erbsland/CpCount.hpp>
#include <erbsland/StandardStreams.hpp>
#include <erbsland/U32String.hpp>

#include <span>

namespace demo {

/// Carry a small byte value through a text field and recover exactly the original bytes.
///
/// Base-N encoding changes the representation of a `ByteBlock`, not its contents. The encoder and decoder use the
/// same format so the text can cross a text-only boundary without treating the bytes as Unicode characters.
void roundTripSoundTag() {
    // Treat the sound label as bytes that need to pass through a text field.
    const auto original = el::ByteBlock::fromSpan(std::span<const char>{"kaiku", 5U});
    const auto format = el::base_n::BaseNFormat::base64();
    const auto encoded = el::base_n::BaseNEncoder{original, format}.toString();

    // Decode with the same format and compare the recovered bytes.
    const auto recovered = el::base_n::BaseNDecoder{encoded, format}.toDataOrThrow();
    el::io::printLine("Text field: "_el, encoded);
    el::io::printLine("Bytes recovered: "_el, recovered == original);
}

/// Encode a small binary sound label as portable Base64 text.
///
/// `BaseNEncoder` consumes a `ByteBlock` and a matching `BaseNFormat`. The default format is canonical, padded
/// Base64. Choose a specific format factory when the receiving protocol requires another alphabet or layout.
void encodeSoundTag() {
    // Encode the UTF-8 bytes of a short Finnish sound label.
    const auto bytes = el::ByteBlock::fromSpan(std::span<const char>{"kaiku", 5U});
    const auto encoded = el::base_n::BaseNEncoder{bytes}.toString();
    el::io::printLine("Base64: "_el, encoded);

    // The same encoder can produce UTF-16 and UTF-32 text when that is the destination's string width.
    const auto encoder = el::base_n::BaseNEncoder{bytes};
    el::io::printLine("UTF-16 characters: "_el, encoder.toU16String().characterLength().toSizeT());
    el::io::printLine("UTF-32 characters: "_el, encoder.toU32String().characterLength().toSizeT());
}

/// Decode Base64 with an explicit byte limit and choose how to handle invalid input.
///
/// `toData()` returns an empty optional for malformed input or an exceeded size limit. `toDataOrThrow()` preserves
/// the distinction as a parse or range exception. Both methods check the decoded byte count before producing data.
void decodeSoundTag() {
    // Decode a small value with the largest byte count this field permits.
    const auto decoded = el::base_n::BaseNDecoder{"a2Fpa3U="_el}.toData(el::ByteCount{5U});
    el::io::printLine("Decoded bytes: "_el, decoded->length().toSizeT());

    // A malformed payload and an oversized payload both fail in the optional form.
    const auto invalid = el::base_n::BaseNDecoder{"a2Fpa3U!"_el}.toData();
    const auto tooLarge = el::base_n::BaseNDecoder{"a2Fpa3U="_el}.toData(el::ByteCount{4U});
    el::io::printLine("Malformed accepted: "_el, invalid.has_value());
    el::io::printLine("Oversized accepted: "_el, tooLarge.has_value());
}

/// Choose a predefined Base-N alphabet or supply a distinct 16-, 32-, or 64-character alphabet.
///
/// Each alphabet character represents four, five, or six bits. The encoder and decoder must use the same alphabet:
/// Base64 and Base64url, for example, assign different characters to their last two values.
void chooseAlphabet() {
    const auto bytes = el::ByteBlock{el::Byte{0xfbU}, el::Byte{0xffU}};
    const auto base64 = el::base_n::BaseNFormat::base64();
    const auto url = el::base_n::BaseNFormat::base64Url();
    el::io::printLine("Base64: "_el, el::base_n::BaseNEncoder{bytes, base64}.toString());
    el::io::printLine("Base64url: "_el, el::base_n::BaseNEncoder{bytes, url}.toString());

    // A custom Base16 alphabet changes the text representation of the same bytes.
    const auto custom = el::base_n::BaseNFormat{el::U32String{U"FEDCBA9876543210"_el}};
    el::io::printLine("Custom Base16: "_el, el::base_n::BaseNEncoder{bytes, custom}.toString());
    el::io::printLine("Bits per character: "_el, custom.bitsPerCharacter());
}

/// Configure the padding character separately from the policy for emitting and requiring it.
///
/// A padding character must be outside both the alphabet and the ignored whitespace set. A custom alphabet starts
/// without padding; assign a character before enabling either padding flag.
void choosePadding() {
    const auto bytes = el::ByteBlock::fromSpan(std::span<const char>{"kaiku", 5U});
    auto format = el::base_n::BaseNFormat::base64();
    format.setPadding(el::Char{U'~'});
    const auto encoded = el::base_n::BaseNEncoder{bytes, format}.toString();
    el::io::printLine("Alternate padding: "_el, encoded);
    el::io::printLine("Round trip: "_el, el::base_n::BaseNDecoder{encoded, format}.toDataOrThrow() == bytes);
}

/// Select exactly which characters a decoder may ignore between Base-N digits.
///
/// Standard formats accept ASCII whitespace. Replacing that set can make a protocol-specific separator legal while
/// ordinary spaces become invalid. The alphabet, padding, and whitespace sets must remain disjoint.
void chooseWhitespace() {
    auto format = el::base_n::BaseNFormat::base64();
    format.setWhitespace(el::CharSet{U'~'});
    el::io::printLine("Tilde accepted: "_el, el::base_n::BaseNDecoder{"a2Fp~a3U="_el, format}.toData().has_value());
    el::io::printLine("Space accepted: "_el, el::base_n::BaseNDecoder{"a2Fp a3U="_el, format}.toData().has_value());
}

/// Change the three independent Base-N behavior flags.
///
/// `EmitPadding` changes encoded output, `RequirePadding` changes decoding policy, and `WrapLines` changes output
/// layout. `setFlags()` replaces all flags; `addFlags()` and `clearFlags()` adjust an existing format.
void chooseFlags() {
    const auto bytes = el::ByteBlock::fromSpan(std::span<const char>{"kaiku", 5U});
    auto format = el::base_n::BaseNFormat::base64();
    format.clearFlags(el::base_n::BaseNFormatFlag::EmitPadding | el::base_n::BaseNFormatFlag::RequirePadding);
    const auto unpadded = el::base_n::BaseNEncoder{bytes, format}.toString();
    el::io::printLine("Unpadded: "_el, unpadded);
    el::io::printLine("Unpadded accepted: "_el, el::base_n::BaseNDecoder{unpadded, format}.toData().has_value());
    format.addFlags(el::base_n::BaseNFormatFlag::RequirePadding);
    el::io::printLine("Padding now required: "_el, el::base_n::BaseNDecoder{unpadded, format}.toData().has_value());
}

/// Accept an unpadded Base64 field only when its protocol allows that spelling.
///
/// Clearing `RequirePadding` permits a missing final padding character. Padding that is present must still be
/// canonical, so adding an incorrect number of padding characters does not become valid.
void decodePaddingPolicy() {
    auto format = el::base_n::BaseNFormat::base64();
    el::io::printLine(
        "Default accepts a2Fpa3U: "_el, el::base_n::BaseNDecoder{"a2Fpa3U"_el, format}.toData().has_value());

    format.clearFlags(el::base_n::BaseNFormatFlag::RequirePadding);
    el::io::printLine(
        "Optional padding accepts a2Fpa3U: "_el, el::base_n::BaseNDecoder{"a2Fpa3U"_el, format}.toData().has_value());
    el::io::printLine(
        "Incorrect padding accepted: "_el, el::base_n::BaseNDecoder{"a2Fpa3U=="_el, format}.toData().has_value());
}

/// Wrap encoded output by setting a line length, separator, and `WrapLines` flag.
///
/// The line length counts encoded characters, including padding. Each separator character must also be accepted
/// whitespace during decoding. Wrapping inserts no separator after the final line.
void wrapLines() {
    const auto bytes = el::ByteBlock::fromSpan(std::span<const char>{"kaiku", 5U});
    auto format = el::base_n::BaseNFormat::base64();
    format.setLineLength(el::CpCount{4U})
        .setLineSeparator(el::U32String{U"\n"_el})
        .addFlags(el::base_n::BaseNFormatFlag::WrapLines);
    const auto wrapped = el::base_n::BaseNEncoder{bytes, format}.toString();
    el::io::printLine("Wrapped:\n"_el, wrapped);
    el::io::printLine("Round trip: "_el, el::base_n::BaseNDecoder{wrapped, format}.toDataOrThrow() == bytes);
}

}