Write a New Data Source

This guide adds a reusable input source to the maintenance application and connects it to a command. Start with Build and Test the Maintenance Tool and read Shared Data Sources for the source lifecycle.

Use tool/src/tool/psl/PublicSuffixSource.hpp and its implementation as a small example. tool/src/tool/unicode/UnicodeSource.hpp shows a source that prepares several related data structures.

1. Define the Source

Derive a default-constructible class from tool::frame::DataSource. Place it in the namespace and source directory for its dataset, following the layout described in Maintenance Framework Overview. Keep construction free of file loading and application-service access; those services are available during callbacks.

Provide a public static sourceIdentifier() returning String. Choose lowercase ASCII letter groups separated by single underscores, such as public_suffix. The identifier must be unique across sources and commands; rendering is reserved. It also names the source’s configuration section.

Implement the required initialize() callback. The base class already provides defaults for a source without configuration: no rules and a no-op configure().

2. Define and Read Configuration

For a configurable source, override configurationRules() to describe settings relative to its section. Add a matching section to an .elcl file under tool/conf/source. The main configuration already includes files in that directory.

Override configure(configuration) to read the validated values and store the settings needed for loading. Use frame::app().files().checked() to check configured project paths. PublicSuffixSource::configure() demonstrates this for an input file. The protected log() stream is available during configuration and initialization.

The framework validates with rule version 1 and applies defaults before calling configure(). Even if every setting has a default, a source with rules requires a regular section. If your source needs no configuration, keep the base defaults and do not add a section. See Configuration Loading and Includes for the validation contract.

3. Load and Validate Inputs

Perform loading in initialize() using frame::app().files(). Use readText() for tolerant UTF-8 or readData() when exact bytes matter. Text reads handle an optional BOM and retain the remaining UTF-8 bytes without validating the complete input. The file service keeps inputs inside the selected project. See Project File Access for its path checks.

Validate the dataset before exposing prepared data. For a pinned upstream text snapshot, verify its manifest, line-ending-independent hashes, and applicable version headers before parsing it. PublicSuffixSource demonstrates these checks; UnicodeSource delegates dataset verification to UcdDataset. Keep acquisition and replacement of upstream snapshots separate from initialization.

Retain source locations while parsing so later validation can identify the failing file and line. Report progress with log() and attach source context to failures using the diagnostic conventions. The application marks the source ready only when initialize() returns successfully.

4. Expose Prepared Data

Provide const accessors for the verified text or prepared structures your commands need. Keep ownership in the source and return immutable references for shared data. Guard access until preparation has completed, as PublicSuffixSource::requireInitialized() does.

Keep output-specific work in the command. For example, the Public Suffix source supplies verified text, while the generator parses rules and encodes its output. The Unicode source also prepares normalization and mapping data because both generators share those inputs.

5. Build and Register the Source

Add the new header and implementation to erbsland-core-tool-lib through the dataset directory’s CMakeLists.txt. For a new directory, add it with add_subdirectory() in tool/src/tool/CMakeLists.txt. The existing tool/src/tool/psl/CMakeLists.txt shows how to include a source alongside its command.

Include your source header in tool/src/main.cpp and register it with app.registerSource<YourSource>() before app.run(). Registration constructs an instance owned by the application and records its static identifier. Commands and sources may be registered in either order.

6. Declare and Use the Dependency

Override the consuming command’s usedDataSources() and return the required static identifiers. For example, IDNA generation declares its Unicode source:

auto IdnaTool::usedDataSources() const -> el::StringList {
    return el::StringList{unicode::UnicodeSource::sourceIdentifier()};
}

List every source the command needs. The application configures all selected sources before initializing any source. It then initializes them in declaration order and initializes repeated identifiers only once. Source configure() callbacks must not access other sources. The Public Suffix command lists both UnicodeSource and PublicSuffixSource.

In the command’s initialize() or execute() callback, retrieve the prepared instance with frame::app().source<YourSource>() and read its accessors. Typed lookup checks readiness; it never starts initialization implicitly.

7. Verify the Source

Add tests under tool/test/unittest/src and include them in that directory’s CMake sources. Cover valid input, invalid configuration, missing files, malformed data, and integrity or version mismatches relevant to your format. Verify that access before successful preparation fails and that a command cannot run after source initialization fails.

Use PublicSuffixSourceTest or UnicodeSourceTest as examples for dataset-specific tests. DataSourceTest already covers shared registration and callback ordering. Tests using frame::app() need an active application; ProjectFixture shows how to provide it.

Follow Build and Test the Maintenance Tool to run the tool tests, then try a consuming command with the Debug executable. Document the source’s settings in Shared Data Sources and explain snapshot maintenance in the usage section when needed.