Development

This section is for developers extending runtime behavior, class logic, and APIs. The core pattern is progressive layering through interface.scm and the class modules loaded at boot.

Architecture

This section explains the implementation layers in the same order they are assembled at runtime. If you are debugging a behavior issue, following this order usually narrows root cause quickly.

System

At runtime, each query is evaluated against the current *sync-state* root. The left side carries executable logic and the right side carries persistent state. The general stack boot flow is:

Verify that the journal state is empty and install root.scm.
Compile the shared object protocol from standard.scm.
Install class definitions for Chain, Tree, Ledger, Federation, and Authorization.
Instantiate Ledger, Federation, and Authorization with distinct durable state and responsibilities.
Store the independently rotatable Interface credential in private Root state, then install generic query/step composition and atomic Root persistence. Root authentication remains host-only.

Concurrency behavior in the journal is optimistic:

queries run against a snapshot of root state
if state changed concurrently, evaluation is retried
writes eventually synchronize through compare-and-set root updates
a global lock is used after repeated collisions to serialize update attempts

Operationally, this behaves like many concurrent readers with controlled commit contention on writes. That model is important for developers writing side-effecting logic: deterministic behavior depends on understanding retry semantics.

Memory/persistence modes:

in-memory mode (default when no --database path is supplied)
persistent RocksDB-backed storage when --database is configured

Periodic stepping:

configured by --step and --period
used in the compose stack to invoke *step* continuously
the outer step is non-mutating: it waits for the retryable internal commit and, only when that commit created a new index, issues one detached call! for (*state* *periodic*); this prevents unchanged steps and optimistic retries from duplicating an index launch

Language

The evaluator embeds s7 Scheme and removes high-risk primitives (file/port/loader/escape surfaces). It also adds convenience and Synchronic primitives. The result is a constrained but expressive execution environment intended for programmable policy and state logic.

References:

The s7 reference is in journal/external/s7/README.md, and the removed primitive list is defined in journal/src/evaluator.rs (REMOVE).

Extended evaluator primitives (examples):

Representative examples include expression->byte-vector and byte-vector->expression, hex-string->byte-vector and byte-vector->hex-string, and utility helpers such as random-byte-vector, time-unix, and print.

Structure

Everything is represented as a hash-addressed DAG of sync nodes. sync-cons, sync-car, sync-cdr, sync-cut, and sync-digest expose low-level structure operations. The language runtime controls interpretation of this DAG through object methods and query handlers. This separation of structure and interpretation is one reason the stack can evolve quickly without breaking core data guarantees.

Objects

Universal object shape:

The left child holds code (constructor/behavior), and the right child holds object state.

standard.scm implements a compact object protocol:

standard.scm implements method dispatch via (object 'method), state access shorthand via (self '(path ...)), composition without inheritance by nesting objects in state, and explicit deep operations such as deep-get, deep-set!, deep-call!, and deep-merge!.

In practice, most domain features are built by composing these object primitives rather than introducing new global runtime behavior.

Specifications

Use this section as a reference when wiring local environments, reviewing pull requests, or mapping code changes to runtime behavior.

Journal Configuration

journal-sdk command-line options:

--database / -d: persistent DB path
--port / -p: HTTP port (default 4096)
--boot / -b: boot expression evaluated at startup
--evaluate / -e: one-shot query and exit
--step / -s: periodic step expression
--period / -c: seconds between step calls

s7 Extended Primitives

The runtime extends base s7 with utility and Synchronic primitives. These signatures are sourced from primitive registrations in journal/src/evaluator.rs and journal/src/lib.rs.

Utility Primitives

Primitive	Signature	Description
`expression->byte-vector`	`(expression->byte-vector expr)`	Encode expression into byte-vector form.
`byte-vector->expression`	`(byte-vector->expression bv)`	Decode byte-vector into expression.
`hex-string->byte-vector`	`(hex-string->byte-vector str)`	Convert hex string to byte-vector.
`byte-vector->hex-string`	`(byte-vector->hex-string bv)`	Convert byte-vector to hex string.
`random-byte-vector`	`(random-byte-vector length)`	Generate securely random byte-vector of given length.
`time-unix`	`(time-unix)`	Return current Unix time in seconds.
`print`	`(print obj ...)`	Print values and return last value.

Sync Structure Primitives

Primitive	Signature	Description
`sync-stub`	`(sync-stub digest)`	Create a stub node from digest.
`sync-hash`	`(sync-hash bv)`	Compute SHA-256 digest of byte-vector.
`sync-node`	`(sync-node digest)`	Load sync node identified by digest.
`sync-node?`	`(sync-node? obj)`	Check whether object is a sync node.
`sync-null`	`(sync-null)`	Return null sync node.
`sync-null?`	`(sync-null? sp)`	Check whether a sync-node or byte-vector is the null sync node.
`sync-pair?`	`(sync-pair? sp)`	Check whether a sync-node or byte-vector is a pair node.
`sync-stub?`	`(sync-stub? sp)`	Check whether a sync-node or byte-vector is a stub node.
`sync-digest`	`(sync-digest value)`	Return digest of a sync-node or byte-vector.
`sync-cons`	`(sync-cons first rest)`	Construct sync pair node.
`sync-car`	`(sync-car pair)`	Return first child of sync pair.
`sync-cdr`	`(sync-cdr pair)`	Return second child of sync pair.
`sync-cut`	`(sync-cut node)`	Convert a sync-node into stub form.

Record Lifecycle Primitives

Primitive	Signature	Description
`sync-create`	`(sync-create id)`	Create record for 32-byte ID.
`sync-delete`	`(sync-delete id)`	Delete record for 32-byte ID (except root record).
`sync-all`	`(sync-all)`	List all record IDs.

Network, Execution, and Crypto Primitives

Primitive	Signature	Description
`sync-call`	`(sync-call query blocking? id)`	Evaluate query against target record (or current record if `id` omitted).
`sync-eval`	`(sync-eval node)`	Instantiate code carried by a sync node in the caller’s current environment.
`sync-let`	`(sync-let ((name value) ...) body ...)`	Evaluate shared self-coded computation in an isolated capability environment with copied inert bindings/results.
`sync-http`	`(sync-http method url . data)`	Perform HTTP request (`get` or `post`).
`sync-remote`	`(sync-remote url data)`	Perform remote post request with payload.
`crypto-generate`	`(crypto-generate seed)`	Derive public/private key pair from seed bytes.
`crypto-sign`	`(crypto-sign private-key message)`	Sign message with private key.
`crypto-verify`	`(crypto-verify public-key signature message)`	Verify signature against message/public key.

sync-let accepts copied ordinary inert data and immutable sync nodes, rejects executable/environment values at the boundary, and blocks ambient journal/root/network/time/random capabilities. Shared self-coded behavior should execute inside sync-let regardless of provenance; installed journal host plumbing executes normally and must not evaluate supplied code in its host environment.

When adding new primitives, keep conversion behavior and security constraints in mind so JSON/Scheme workflows remain predictable.

Standard Objects

The object model is loosely Python-inspired:

The model uses methods instead of free global variables, captures a single state root per object, uses class-like definitions via define-class, and favors explicit composition over inheritance.

State path shorthand uses cdr/car traversal semantics:

State path shorthand follows cdr/car traversal semantics such as (self '(1 0 0 1 ...)).

In the tables below, methods prefixed with * (for example *init*) are still part of the public class API. Only methods prefixed with ~ are treated as internal/helper methods.

Normative Guidelines

For standard object implementations, define-class and define-method are the normative authoring forms.

New runtime classes SHOULD be expressed as a single define-class form.
Class members MUST be declared with define-method; arbitrary top-level expressions inside a class body are not part of the supported model.
Constructors SHOULD be implemented as (*init* self ...) methods rather than ad hoc initialization outside the class form.
Internal helper behavior SHOULD be exposed as ~-prefixed methods, while stable external behavior SHOULD use non-~ method names.

This convention keeps class loading deterministic, keeps object serialization semantics predictable, and aligns with how standard.scm validates class definitions during make.

Standard objects dispatch methods directly and restore their prior state when a method raises an error. Direct sync-eval is raw object loading, not a containment boundary. Installed Root, Interface, Standard, Ledger, Federation, and Authorization are trusted host infrastructure. Standard’s local loader supplies trusted generated code with stored durable state; embedded Tree, Chain, custom, peer, and historical behavior runs through child-only sync-let boundaries.

Standard Class

Public API (standard.scm):

Method	Signature	Description
`make`	`(make self class)`	Compile an uninitialized object node from a `define-class` form.
`init`	`(init self class . arguments)`	Compile an object, invoke `init` when present, and return its initialized node.
`deep-get`	`(deep-get self object path)`	Read value/object across nested object boundaries.
`deep-set!`	`(deep-set! self object path value)`	Write value across nested object boundaries.
`deep-slice!`	`(deep-slice! self object path)`	Slice object graph along path while preserving digest invariants.
`deep-prune!`	`(deep-prune! self object path)`	Prune object graph along path while preserving digest invariants.
`deep-merge!`	`(deep-merge! self object-source object-target)`	Merge digest-equivalent object structures.
`deep-copy!`	`(deep-copy! self object path-source path-target)`	Copy value from one nested path to another.
`deep-call`	`(deep-call self object path function)`	Call a function at a nested path without rebuilding parent state.
`deep-call!`	`(deep-call! self object path function)`	Call a function at a nested path and persist resulting state.
`serialize`	`(serialize self node query)`	Build compact proof-oriented serialization using request-local primitive tracing.
`deserialize`	`(deserialize self serialization)`	Rebuild sync-node structure from serialization output.

Tree Class

Public API (tree.scm):

Method	Signature	Description
`obj->node`	`(obj->node self obj)`	Encode Lisp/runtime value into sync-node storage representation.
`node->obj`	`(node->obj self node)`	Decode sync-node storage representation into runtime value.
`get`	`(get self path)`	Read value at key-path; returns value, `(nothing)`, `(unknown)`, or a directory listing.
`equal?`	`(equal? self source path)`	Exact structural equality check between two paths.
`equivalent?`	`(equivalent? self source path)`	Digest-equivalence check between two paths.
`set!`	`(set! self path value)`	Write value at a nonempty path, delete with `(nothing)`, or clear the complete map with `(set! self '() '(nothing))`; the empty path cannot store a scalar or object.
`copy!`	`(copy! self source path)`	Copy source to target; a missing source applies normal `(nothing)` deletion semantics, and only a directory source may replace the complete map at the empty path.
`prune!`	`(prune! self path keep-key?)`	Prune proof/state detail at path.
`slice!`	`(slice! self path)`	Slice state to keep proof for path and cut unrelated branches.
`merge!`	`(merge! self other)`	Merge compatible tree structures.
`valid?`	`(valid? self)`	Validate structural/key-prefix consistency of the tree.

Linear Chain Class

Public API (linear-chain.scm):

Method	Signature	Description
`init`	`(init self)`	Initialize empty chain state.
`get`	`(get self index)`	Return entry at normalized index.
`previous`	`(previous self index)`	Build proof chain ending at index.
`digest`	`(digest self (index ...))`	Return digest for proof chain at index.
`size`	`(size self)`	Return chain length.
`index`	`(index self index~)`	Normalize/index-check external index input.
`push!`	`(push! self data)`	Append entry to chain.
`set!`	`(set! self index data)`	Replace entry at index.
`slice!`	`(slice! self index)`	Slice proof view around index.
`prune!`	`(prune! self index)`	Prune proof detail at index.
`truncate!`	`(truncate! self index)`	Hide entries through the inclusive index while preserving the chain digest.

Log Chain Class

Public API (log-chain.scm):

Method	Signature	Description
`init`	`(init self)`	Initialize empty log-structured chain state.
`size`	`(size self)`	Return chain length.
`index`	`(index self index~)`	Normalize/index-check external index input.
`get`	`(get self index)`	Return entry at normalized index.
`previous`	`(previous self index)`	Build proof chain ending at index.
`digest`	`(digest self (index ...))`	Return digest for proof chain at index.
`push!`	`(push! self data)`	Append entry to chain.
`set!`	`(set! self index data)`	Replace entry at index.
`slice!`	`(slice! self index)`	Slice proof view around index.
`prune!`	`(prune! self index)`	Prune proof detail at index.
`truncate!`	`(truncate! self index)`	Hide entries through the inclusive index while preserving the chain digest.

Ledger Class

Public API (ledger.scm):

Method	Signature	Description
`init`	`(init self standard config tree-class chain-class)`	Initialize ledger with standard helper, inline config expression, and storage classes.
`config`	`(config self (path '()))`	Return allowed public/operational configuration; reject private Ledger paths.
`descriptor`	`(descriptor self index)`	Return the public descriptor for a selected local state.
`size`	`(size self)`	Return permanent chain length.
`read`	`(read self index supplied-object)`	Anchor an exact partial chain against local history.
`get`	`(get self path)`	Read a staged Tree-native value.
`get-batch`	`(get-batch self paths)`	Read ordered staged values from one Ledger snapshot.
`set!`	`(set! self path value expected? expected)`	Stage a byte-vector write/deletion, optionally after an exact snapshot comparison.
`set-batch!`	`(set-batch! self changes)`	Validate every `(path value [expected])`, compare expectations against one snapshot, and atomically apply replacements in order.
`resolve`	`(resolve self path pinned? proof? head ancestor?)`	Resolve committed content with optional retention, proof, prepared-head, and ancestor projection context.
`resolve-batch`	`(resolve-batch self paths pinned? heads ancestors proof?)`	Resolve ordered committed paths and optionally produce one union proof for a compatible group.
`trace`	`(trace self path head)`	Serialize a proof from local history or a prepared head.
`trace-batch`	`(trace-batch self paths head)`	Serialize one union-of-accesses proof for same-anchor paths.
`pin!`	`(pin! self path proof)`	Pin a local path, optionally from a prepared proof response.
`pin-batch!`	`(pin-batch! self paths proofs)`	Atomically pin ordered paths; shared prepared proofs reuse one anchored immutable head.
`unpin!`	`(unpin! self path)`	Remove a previously pinned path.
`unpin-batch!`	`(unpin-batch! self paths)`	Atomically apply ordered digest-preserving proof cuts.
`pinned?`	`(pinned? self path)`	Test whether permanent retention contains a path.
`signed-head`	`(signed-head self known-index)`	Return a signed synchronization head.
`peer-head`	`(peer-head self alias index)`	Return committed peer evidence or an opaque peer checkpoint.
`merge-head!`	`(merge-head! self alias supplied-head)`	Merge digest-equivalent fetched proof material transiently.
`store-peer-head!`	`(store-peer-head! self alias verified-head)`	Apply Federation-verified peer evidence.
`delete-peer-head!`	`(delete-peer-head! self alias)`	Retire active peer evidence while preserving identity continuity.
`step!`	`(step! self prepared-inputs)`	Commit staged state, sign it, and apply retention.
`update-config!`	`(update-config! self changes)`	Apply one validated configuration change.

Federation Class

federation.scm owns peer operational configuration, networking, exact-route construction, signed invocation transport/authentication, and synchronization continuations. It is installed host infrastructure, not a transferred durable peer object or an external API independent of Interface.

Method	Signature	Description
`config`	`(config self (path '()))`	Read allowed Federation configuration.
`update-config!`	`(update-config! self changes)`	Apply validated peer/acceptance configuration.
`peers` / `peer`	`(peers self)` / `(peer self alias)`	Return inert operational peer summaries.
`bridge!`	`(bridge! self ledger alias endpoint)`	Establish or resume a reciprocal bridge through the two-phase continuation protocol.
`delete-bridge!`	`(delete-bridge! self ledger alias)`	Delete an active relationship while retaining identity continuity.
`synchronize!`	`(synchronize! self ledger request)`	Verify and accept a reciprocal signed-head request.
`bridge-synchronize!`	`(bridge-synchronize! self ledger alias)`	Prepare initiator-side synchronization.
`step!`	`(step! self ledger)`	Return synchronization work for the permanent initiator role.
`route`	`(route self ledger request)`	Construct exact committed terminal routing material.
`invoke`	`(invoke self ledger operation arguments route history identity)`	Sign, deliver, and verify federated `get`, `set!`, `get-batch`, `set-batch!`, or `resolve`.
`authenticate`	`(authenticate self ledger invocation)`	Anchor and authenticate a terminal invocation before local authorization.

Authorization Class

authorization.scm owns policy only. Interface owns authentication order and passes canonical principals and terminal-local history context into policy.

Method	Signature	Description
`authorizations`	`(authorizations self (principal #f))`	List rules visible to an owning local principal or administrator.
`authorize!`	`(authorize! self rule)`	Add one validated path-scoped rule.
`deauthorize!`	`(deauthorize! self rule-or-selector)`	Remove an exact rule or bridge-prefix selection.
`authorized?`	`(authorized? self principal key-index path operation)`	Return `direct`, `ancestor`, or `#f` for grantable `get`, `set!`, or `resolve` operations. `call!` authority is enforced only by Interface administrator membership.

Testing

Testing coverage is intentionally split across correctness, stress, and topology concerns so regressions can be isolated by failure type.

Synchronous Testing

Use records/tests for deterministic, script-driven correctness checks over simulated journals.

Prerequisites:

You need a Rust toolchain and the Journal’s C build dependencies. The Records runner is cross-platform and does not require a POSIX shell or a separately installed Journal SDK executable.

Run from the repository root:

CARGO_TARGET_DIR=journal/target \
  cargo build --release --manifest-path records/tests/Cargo.toml
journal/target/release/records-test --suite records/tests/suite.toml

These tests validate class behavior in fresh isolated evaluator processes. The same suite also runs deterministic cross-journal Interface cases under records/tests/interface.

Service Stack Smoke Tests

Use tests/api/local-compose.sh when you need to validate the integrated runtime plus the primary operator surfaces:

/explorer
/workbench
/docs
/api/v1/...
the WebDAV-backed file-system projection

Prerequisites:

You need a Compose-compatible container runtime (Docker Compose, Podman Compose, or podman-compose) and curl for route and API checks.

Run interactive stack (from repo root):

COMPOSE_PROJECT_NAME=sync-local SECRET=root-password \
INTERFACE_SECRET=interface-password ADMIN_PASSWORD=admin-pass \
HTTP_PORT=8192 HTTPS_PORT=8193 \
tests/api/local-compose.sh up

Run automated health/smoke checks:

tests/api/local-compose.sh smoke

The smoke script verifies route readiness, key API behavior (for example size and authenticated configuration responses), and the compose-integrated file-system service.

Service-Level Tests

services/explorer and services/workbench include client-side unit tests, and services/file-system includes Go tests for WebDAV path projection behavior.

Prerequisites:

You need Node.js 20+ and npm to run the Explorer and Workbench unit test suites.

Run tests:

cd services/explorer
npm install
npm test

cd services/workbench
npm install
npm test

cd services/file-system
go test ./...

Stress Testing

Use tests/load/locust for HTTP load generation against gateway general endpoints.

Prerequisites:

You need a running gateway endpoint (typically from tests/api/local-compose.sh up), Python 3 with pip and Locust dependencies from requirements.txt, and an API token created from /auth/settings or POST /api/v1/tokens.

Install and run:

cd tests/load/locust
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
API_TOKEN=sync-... locust --host=http://localhost:8192

Headless example:

cd tests/load/locust
. .venv/bin/activate
API_TOKEN=sync-... locust --host=http://localhost:8192 --users=10 --spawn-rate=2 --run-time=60s --headless

Multi-Node Compose Testing

Use tests/network/compose when you want a local multi-node network with one full stack and one social agent per node, without using FIREWHEEL.

Prerequisites:

You need a Compose-compatible container runtime, Python 3, and PyYAML available in the environment where you run the generator.

Build images and generate the stack (from repo root):

tests/network/compose/local-compose.sh generate

Run it:

cd tests/network/compose
podman compose up

Generator defaults:

NODE_COUNT=4
SECRET=root-password
INTERFACE_SECRET=interface-password
ADMIN_PASSWORD=admin-pass
CONNECTIVITY=2
PERIOD=2
WINDOW=1024
SIZE=32
ACTIVITY=4 (seconds between controlled activity cycles; 0 runs maximum-throughput saturation traffic)
BATCH unset (optional positive batch size for continuous activity)
WORDS=8

Network model:

each node gets its own private-<n> network
all routers and journals join a shared public network
routers remain the named HTTP boundary between nodes
file-system remains the WebDAV boundary per node through each router’s /webdav/ route
social agents live on public and talk through routers
optional BATCH=N activity preserves the random anchor route, selects N unique authorized paths from that exact route/access group, and uses staged or latest-history batch endpoints; setup and readiness remain scalar
each journal’s fixture assigns about half of SIZE to admin/data/public, then quarters/eighths/etc. to fixed randomly selected exact routes; colon-separated directory labels are farthest-to-nearest
public fixture keys are remotely read-only, private keys are writable only by the admin at the far end of the assigned route, and activity updates those same keys in place
descendant grants admit ordinary ancestor listings, while each listed child remains independently access-controlled

Generated topology artifacts:

compose.yml
peers.json
metrics/social-agent-*/
results/social-agent-*/benchmark.json
results/network-benchmark.json when aggregate_results.py is running

peers.json uses:

nodes: journal-name to router-host mapping
edges: deterministic reciprocal-bridge initiator adjacency using a fixed generator seed and bounded-degree sampling

Port allocation:

router HTTP starts at 8192 and increments by node index
WebDAV is exposed through each router under /webdav/

Metrics output:

each social agent writes Prometheus textfile metrics into its own host-mounted directory under metrics/
HTTP request counters/rates remain distinct from logical path-operation counters/rates, so batching efficiency is visible without relabeling one request as many requests
this preserves the current FIREWHEEL-style agent metric output path for local compose runs without adding a full monitoring stack yet

Benchmark output:

each social agent also writes a rolling JSON snapshot to results/social-agent-*/benchmark.json
these files are the simplest place to start for end-to-end throughput benchmarking; ACTIVITY=0 runs without delay for saturation testing, positive intervals exercise controlled traffic, and ACTIVITY_DISABLED=1 provides a no-background-activity baseline
benchmark dashboards lead with successful logical path operations per second and retain HTTP requests per second alongside them; neither metric claims byte throughput
run python3 aggregate_results.py in the harness directory to maintain results/network-benchmark.json as a cluster-wide aggregate snapshot

For local testing of an unpublished social-agent image, build a local tag and override it during generation:

docker build -t social-agent:dev -f tests/network/common/social-agent/Dockerfile tests/network/common/social-agent

cd tests/network/compose
IMAGE_OVERRIDE_SOCIAL_AGENT=social-agent:dev python3 generate.py
podman compose up

Network Emulation

Use tests/network/firewheel for topology-level distributed simulation across journals, agents, and monitoring components.

Prerequisites:

You need the FIREWHEEL framework installed and configured, Docker available on the host, and enough host resources for multi-node simulation workloads.

Example experiment:

cd tests/network/firewheel
firewheel experiment -r synchronic_web.ledger_journal:4:2 synchronic_web.social_agent:4:32:2:8 synchronic_web.network_monitor control_network minimega.launch

This mode is best for validating emergent behavior, peer dynamics, and monitoring instrumentation under realistic network layouts.

Guidance

This section captures conventions and failure patterns that are useful during day-to-day implementation and review work.

Style Guide

Recommended conventions used across current class modules:

prefer (object 'method) style to reduce namespace pollution
use *variable* names for special/global runtime variables
use ~name for internal/helper methods and hidden fields
use association lists for web-facing calls to keep JSON conversion predictable
keep code in left node and mutable state in right node

Following these conventions substantially improves maintainability when multiple contributors are updating class logic concurrently.

Gotchas

Common pitfalls in this stack:

read remote moving state -> wait on network -> mutate local state based on stale assumptions
mutating a child object does not automatically reattach it into its parent container; write it back explicitly
broad *eval*/*call* usage can bypass intended API boundaries; prefer narrow method-level updates where possible

When possible, add targeted assertions around state digests or path expectations to detect these issues early during development.