Jahan — Engine Architecture¶

Scope: the open-source simulation engine, implementation only. No business, market, positioning, pricing, or tiering content — this document is what you build from. Language/runtime: Python 3.12+, uv-managed. License: Functional Source License 1.1, Apache-2.0 future license (LICENSE.md). Code salvaged from upstream keeps its own license and attribution; see SALVAGE.md. Salvage sources: OASIS (Apache-2.0), ASAL (Apache-2.0), MatrAIx-Persona-8B (MIT) + HF dataset MatrAIx2026/MatrAIx_Persona_1M. Companion documents: CONTEXT.md is the project glossary and is authoritative on naming — where a word here disagrees with it, the glossary wins. docs/adr/ records decisions whose reasoning would otherwise be invisible in the code. docs/prd/ and plans/ carry per-module requirements and phasing.

1. What the engine does¶

A product brief goes in. A population is sampled from a real 1,290-attribute persona dataset and given a social structure. Those personas are run through interacting environments — survey room, social feed, forum, word-of-mouth — where they see each other's behavior. Purchase intent is elicited as free text and converted to a Likert distribution by embedding similarity, never by asking a model for a number. Everything that happens is written to an append-only trace. A report is derived from that trace, and no claim renders without resolving to trace IDs.

flowchart LR
    A["brief.yaml"] --> B["brief"]
    B --> C["population<br/>personas + graph + communities"]
    C --> D["runner"]
    D --> E["world<br/>survey · feed · forum · WOM"]
    E <--> F["agent<br/>context · memory · turn"]
    F --> G["elicitation<br/>SSR → Likert PMF"]
    F --> H["inference<br/>batch queue · limiter · cache · fake mode"]
    G --> H
    E --> I["trace<br/>append-only spine"]
    F --> I
    I --> J["analysis<br/>digests · clusters · anomalies · findings"]
    J --> K["report<br/>render only"]
    D --> J

2. Architectural principles¶

These are the rules that decide where code goes. They are binding; §13 records where the design previously violated them and what changed.

P1 — Deep modules. A module earns its existence by hiding more than it exposes. If a module's interface is about as complex as its implementation, it should be part of its only caller. Measured concretely: a module with one caller and an interface of 3+ parameters carrying intermediate types is a merge candidate, not a module.

P2 — No orphan intermediates. A type that exists only to travel between two modules, and that no external caller ever wants on its own, is evidence the boundary is in the wrong place.

P3 — One owner per invariant. Every cross-cutting rule (budget, persona conditioning, provenance, trust) has exactly one module that owns and enforces it. Rules enforced in three places are enforced in none.

P4 — Ports at every non-deterministic boundary. Anything with network, filesystem, clock, or model non-determinism is reached through a port with at least two adapters: the real one and an in-memory one used by tests. This is what makes deep modules testable without mocking internals.

P5 — Boundary tests are the primary test layer. Property tests guard data invariants; golden runs detect drift. Correctness is asserted at deep-module boundaries, against observable outputs, using in-memory adapters.

P6 — Salvaged code stays diffable. Forked OASIS files keep upstream file structure and license headers so quarterly upstream diffs remain mechanical. Salvage is isolated behind our own port so upstream shape never leaks into engine-wide interfaces.

3. Module map¶

Fourteen modules. Every one is either a leaf contract, a deep behavioral module, or a thin adapter — over salvaged code, or over the modules that own the numbers.

#	Module	Owns	Hides	Interface
1	`schemas`	every type crossing a module boundary	nothing (leaf, zero logic)	types + validators
2	`brief`	intake, category ontology, assumption ledger	YAML parsing, evidence fetching, ontology versioning	`load_brief(path, ontology_dir) -> BriefPack`
3	`population`	who is in this study and how they are connected	4-bit decode, conditioning filter, postings filter, audience-proportional sampling, distribution gates, sparse completion, graph generation, Leiden community detection	`build(brief, n, population_seed) -> Population`
4	`inference`	every model call in the system	provider routing, retries, coalescing, caching, token accounting, model pinning, fake mode	`chat(role, msgs) -> Completion`, `embed(texts) -> Vectors`
5	`elicitation`	free text → Likert PMF (SSR)	anchor sets, reference-set averaging, τ, non-collapse checks	`score(text) -> SsrResult`
6	`agent`	one persona's reaction to one impression	context assembly, persona conditioning, memory retrieval, reflection, tier routing, output parsing, guardrails, prompt reconstruction	`turns(jobs) -> outcomes`
7	`world`	environment mechanics and who sees what	platform state, action handling, recsys ranking, activation clock, interventions	`reset(header) -> WorldDelta`, `step(tick, turns) -> WorldDelta`
8	`runner`	executing a study within a budget	job expansion, worker pool, checkpointing, resume, budget governance and its enforcement, sweep, progress publication	`run(RunConfig) -> RunResult`
9	`trace`	the append-only record and its read views	SQLite→Parquet lifecycle, partitioning, registry, query shapes	`write(events)`, `view(run_id) -> TraceView`
10	`analysis`	deriving meaning from a trace	digest computation, verbatim clustering, anomaly detection, finding authorship, trust guard, trace summary, trajectories, workspace summary	`digest(view) -> OutcomeDigest`, `findings(views) -> list[Finding]`, `trace_summary(views)`, `trajectories(...)`, `workspace_summary(entries)`
11	`report`	rendering	templates, markdown/JSON parity	`render(findings, digests) -> Report`
12	`cli`	entrypoints	argument plumbing, exit codes	`coreset-gate`, `ssr-replica`, `concepts run`, `sweep run`, `ontology check`, `brief check`
13	`web`	the engine over HTTP	serialisation, filtering, paging, run lifecycle as subprocesses	`create_app(...) -> FastAPI`
14	`ui`	the Next.js interface	nothing — it renders what the API returns	the only module not written in Python

Mapping from the previous 28-module numbering¶

Kept so the salvage inventory and any existing notes stay resolvable.

Old	New home	Old	New home
M1	`schemas`	M15	`world`
M2	`brief`	M16	`runner`
M3, M4, M5, M6	`population`	M17	`runner` (budget, single enforcement point)
M7	`inference`	M18	`inference` (cache folded in)
M8, M10, `ContextBuilder`	`agent`	M19	`trace`
M9	`elicitation`	M20	`runner` (sweep) + `analysis` (digest)
M11	deferred (§12)	M21–M23	deferred (§12)
M12, M13, M14	`world` (RetailShelf deferred, §12) — old M-numbers, unrelated to the new §5.13 `web`	M24, M25	`AnchorSource`/`PersonaPatchSource` ports + deferred harness (§12)
—	—	M26	deferred (§12)
—	—	M27	`analysis` (authorship) + `report` (render)
—	—	M28	`cli`

4. Ports and adapters¶

Six ports. Each has a production adapter and an in-memory adapter; the in-memory one is what every boundary test runs against.

Port	Production adapter	Test adapter	Why it's a port
`ChatPort` / `EmbedPort`	one OpenAI-compatible endpoint over `httpx` — a user-run gateway (LiteLLM by default) or vLLM/Ollama directly; no provider SDK (ADR 0021)	`FakeInference` — deterministic completions keyed by prompt hash	network + model non-determinism
`CoresetSource`	`HfCoresetSource` — reads shards a user fetched explicitly, verifying every file against `manifest.json` on each use; downloads nothing	`FixtureCoresetSource` (committed rows), `SyntheticCoresetSource` (generated; its fields state a tier — a shape that declares no assignment type is measured, one standing for a model's reading is extracted, for invented rows synthesized)	filesystem + a 4.17 GB dependency that must not be bundled, and a decoder whose rules are the dataset's knowledge
`CoresetCatalog`	`IndexCoresetCatalog` — coverage and counts answered from postings built once from the cached shards' packed arrays and saved beside the cache, never a row decode (the release's own 2.6 GB `postings.sqlite` is not used)	the same `SyntheticCoresetSource`, which implements both protocols	a four-orders-of-magnitude cost gap and a different failure mode are why `preview()` is typed to the catalog, not the source, so it cannot grow a row read
`AnchorSource`	`PackagedAnchors` — versioned anchor sets and default τ shipped in-repo	`StubAnchors` — fixed vectors, exact expected PMFs	lets anchors and τ be replaced without touching `elicitation`
`PersonaPatchSource`	`NullPatchSource` (ships as default)	`InMemoryPatchSource`	lets externally-derived persona corrections be applied without changing `population`'s signature later
`TraceSink`	`SqliteParquetSink`	`InMemoryTraceSink`	filesystem + a hot write path
`EvidencePort`	`HttpEvidence` — a standard-library GET, http(s) only, capped and timed out	`InMemoryEvidence` — url to bytes, with failures on demand	network, and the only reach outside the engine that a study author triggers

Two of these are load-bearing beyond testability:

CoresetSource and CoresetCatalog are the dataset-independence seam. The engine never requires the 1M-row dataset to be present or bundled, and it never fetches a shard on a user's behalf: HfCoresetSource reads a cache the user populated with an explicit hf download, and refuses an absent shard with the exact command that fetches it, verifying every cached file against the manifest's digest on each use (ADR 0016, ADR 0020). SyntheticCoresetSource makes concepts run --fake work end-to-end with zero downloads and zero API keys, which is the quickstart path. preview() is typed to CoresetCatalog, not CoresetSource, so a pre-flight that only needs counts cannot read a row. Nothing in the engine may import a dataset file path directly — it goes through a port.

AnchorSource and PersonaPatchSource are the calibration seams. The engine ships defaults (a documented default τ, no patches). If anchors, τ, or persona corrections are later derived from real human data, they arrive as a different adapter — no change to elicitation or population signatures. Designing these in now is cheap; adding them after the interfaces are public is a breaking change.

flowchart TB
    subgraph CORE["engine core — pure, deterministic under seed"]
        POP["population"]; AG["agent"]; EL["elicitation"]
        WO["world"]; RUN["runner"]; AN["analysis"]; RP["report"]
    end
    subgraph PORTS["ports"]
        CP["ChatPort / EmbedPort"]; CS["CoresetSource"]
        AS["AnchorSource"]; PP["PersonaPatchSource"]; TS["TraceSink"]
    end
    subgraph PROD["production adapters"]
        P1["OpenAI-compatible HTTP"]; P2["HF hub download"]
        P3["packaged anchors"]; P4["null patches"]; P5["SQLite → Parquet"]
    end
    subgraph TEST["test adapters"]
        T1["FakeInference"]; T2["fixture / synthetic coreset"]
        T3["stub anchors"]; T4["in-memory patches"]; T5["in-memory trace"]
    end
    CORE --> PORTS
    CP --> P1 & T1
    CS --> P2 & T2
    AS --> P3 & T3
    PP --> P4 & T4
    TS --> P5 & T5

5. Module specifications¶

Each spec states what the module owns, what it hides, its interface, its internals, its salvage, and its boundary tests.

5.1 `schemas` — contracts¶

Owns: every type crossing a module boundary, and the invariants that are true of the data by definition. Hides: nothing. This is a leaf. Hard rule: imports nothing from jahan, and only pydantic (with its own pydantic-core engine) + stdlib from outside. No numpy, no pyarrow, no networkx. If schemas needs a project import, a type is in the wrong file.

Interface: ~40 frozen pydantic v2 models with extra="forbid", allow_inf_nan=False, and immutable containers only — tuples, frozensets and FrozenDict; list, dict and set annotations are refused at class definition, and model_copy(update=...) re-validates — grouped one file per domain — brief, persona, population, sim, run, trace, report, enums, errors, base.

Load-bearing validators: - PMF5 — 5 values, all strictly > 0, sum ∈ [0.999, 1.001]. Strict positivity because polarization is a JSD across community PMFs and downstream metrics are KL-family; SSR's softmax cannot emit an exact zero, so a zero means something upstream is broken. - ModelPins — rejects empty, wildcarded and version-floating ids, including -latest, :latest and @latest. This is what makes replayability structural rather than aspirational. - Finding — distinct supporting records and a non-empty disconfirming test. Makes an unprovenanced or auto-crowned claim unconstructible. - TrustStatement — any level above UNCALIBRATED needs a CalibrationRef pinning its benchmark and human study by hash, with measured similarity and rank attainment at or above 0.80 (ADR 0008); nothing in the repository produces one. - Computed values — gate verdicts, the SSR headline mass, whether an exposure was noticed, digest metrics, and every identity hash — serialize but are never trusted on input: a supplied value that disagrees with the computation is refused. - Invariants that span records are enforced by the aggregate that holds them, never by a record labelling itself: Population checks conditioning, field domains, baseline beliefs and gate grounding against the brief pack; Turn checks a reaction is about something shown; TracePartition checks claims, stimuli, personas and pins against its header; Report checks digests and anomalies against its run configuration.

Canonical hashing (base.py): dump the model with mode="json", then walk that dump alongside the live model so each nested model's _hash_exclude_ and _hash_version_ apply at its own depth — a single top-level exclude silently ignores nested declarations. Set-valued fields are sorted (their iteration order depends on the process hash seed), negative zero becomes zero, and the contract version is folded under _schema_version, a key pydantic will not let any field occupy. The result is hashed as sha256(json.dumps(payload, sort_keys=True, separators=(",",":"), allow_nan=False)); sort_keys is required because insertion order must not move a hash. mode="json" means enum member renames don't move hashes, only value changes do. Four hashes: brief_hash, population_hash, config_hash (includes SCHEMA_VERSION), graph_hash. Each is derived from its object — the population hash from its brief, ontology, seed, personas, graph and communities; the graph hash from its ties — and stated only where the object itself does not travel, then verified wherever it is present (ADR 0009). The representative study's identities are pinned per contract version, so a hashed shape cannot change without a version bump.

Performance rule: pydantic guards boundaries, not inner loops. Three paths use model_construct — which refuses raw mappings wherever a field expects a model, so a mistake fails at construction rather than far downstream — and are validated only in fake-mode CI: trace writes (~500k events/world), graph adjacency (~100k edges), per-row coreset decode. Embeddings never live inside a model — EmbeddingRef {model_id, dim, index} points into one contiguous float32 array per population.

Boundary tests: round-trip property tests per type (hypothesis); committed hash-stability golden fixtures; one test per validator; an AST test asserting the leaf rule holds; a forward-compat test that a trace partition written under an older contract version still loads.

5.2 `brief` — intake¶

Owns: turning user YAML into validated structures, the category ontology, and the assumption ledger. Hides: parsing, strict-mode rejection, claim ID assignment, evidence-URL fetch and hashing, ontology file versioning.

Interface: load_brief(path: Path, ontology_dir: Path) -> BriefPack · fetch_evidence(path: Path, port: EvidencePort, *, refetch: bool = False) -> FetchReport · assumptions_of(pack: BriefPack) -> tuple[Assumption, ...]

Internals: yaml.safe_load → ProductBrief.model_validate (unknown keys rejected); claim IDs auto-assigned C1..Cn and never authored; a claim cites evidence_url and the content hash beside it comes from <brief>.evidence.json, written by a separate fetch_evidence command through EvidencePort — loading touches no network and no clock, and a cited URL that was never fetched is refused (ADR 0013); the brief names an ontology version and the category ontology is loaded as versioned JSON from ontology_dir — never embedded in the brief (ADR 0004) — then joined into a BriefPack that refuses a category or version mismatch and checks audience filters against the ontology; the ontology declares a field domain for every attribute it uses, and supplies an attribute relevance order (every declared attribute ranked once, the conditioning set first, so a token budget cuts only non-conditioning attributes), the anchor set for each construct, and completion_policy (which persona fields may be synthesized — economics, decision rules, media yes; demographics, psychographics never); stimulus kinds are one global closed set, not a per-category list; assumption ledger built so every assumption is a first-class record surfaced in the report.

Gotcha to encode: claims are the atomic stimulus unit — feed cards, forum posts and report findings all reference Claim.id. Reordering claims mid-study silently breaks comparability, which is why brief_hash covers claim order and the runner refuses to resume a run whose brief_hash moved. And because the ontology is the study's claim about what a category is, its conditioning set must be checkable against a real source — population.preview answers that from the catalog's coverage, so an ontology that names a conditioning attribute nobody carries fails at authoring, not at the end of the first draw (ADR 0020).

Boundary tests: malformed YAML rejected with a GateFailure, not a stack trace; unknown keys rejected; claim IDs stable and contiguous; brief_hash invariant to comment/whitespace changes and sensitive to claim reordering.

5.3 `population` — who is in the study¶

Absorbs old M3 (loader), M4 (sampling), M5 (projection), M6 (graph).

Owns: producing the complete population for a study — personas, their social graph, their communities, and the report that says whether the sample is acceptable. Hides: packed-4-bit decode, null-bitmask and codebook semantics, attribute_overrides precedence, postings-index filtering, conditioning-set filtering, audience-proportional sampling, categorical and ordinal distribution gates, sparse completion via the model, embedding computation, two-layer graph generation, Leiden community detection, community-card authoring.

Interface: the pre-flight has two stages (ADR 0020) — preview answers what the corpus holds from an index, assess answers whether a draw is sound from rows:

def preview(request: PreviewRequest, *, catalog: CoresetCatalog) -> AudiencePreview

def assess(pack: BriefPack, n: int, population_seed: int, *,
           coreset: CoresetSource,
           parameters: PopulationParameters = PopulationParameters()) -> GateReport

def build(pack: BriefPack, n: int, population_seed: int, *,
          coreset: CoresetSource, inference: ChatPort,
          patches: PersonaPatchSource = NullPatchSource(),
          embed: EmbedPort | None = None,
          parameters: PopulationParameters = PopulationParameters()) -> BuiltPopulation

def interpret_audience(description: str, pack: BriefPack, *,
                       catalog: CoresetCatalog, inference: ChatPort) -> Audience

preview is typed to CoresetCatalog — a separate port from CoresetSource, answering coverage(attributes, sources) and count(predicates, present) from a per-value index, never by decoding a row — because the two answers differ by four orders of magnitude in cost and typing preview to a source is how it would eventually grow a row read. SyntheticCoresetSource implements both; IndexCoresetCatalog is built from HfCoresetSource's packed arrays without decoding a persona. interpret_audience turns words into predicates through ChatPort, bounded by the coverage table (ADR 0014). assess stops after gating and calls no model, so checking a cohort is free; build returns a BuiltPopulation: the validated Population — personas, graph, communities, gate_report, manifest (row ids, requested and achieved mix, seeds, completion provenance, admissible_sources, population_hash) — alongside the embedding array when embeddings are enabled. The array travels beside the contract, never inside it: schemas is a leaf package and can hold no numpy, and a hash over 2,000 × 384 floats would dominate every serialization in the system. EmbeddingRef.index is the join.

Sampling, graph rewiring and community detection draw from independent streams spawned off the one population seed (SeedSequence(population_seed).spawn(3)) rather than from separately authored seeds, which would invite correlated draws if anyone set them equal (ADR 0001). Leiden is randomised, so it is seeded too. A population is built once and carried, never rebuilt: determinism holds under a deterministic inference port, and against a live model the completion of sparse fields makes a rebuild a different population (ADR 0015).

Why these four merged: nobody ever wants a DecodedRow — it existed only to cross a boundary. Field-mapping knowledge was split between decode and projection. The completion policy was defined in brief but enforced in projection. The graph is not a separate concept from the population; a population without its social structure is not usable by any caller. One module, one question: who is in this study?

Internals, in order: 1. Resolve — intersect postings sets per filter key and require every attribute in the category's conditioning set to be populated; index-only, never opens a data shard. Filtering for conditionability here rather than dropping sparse rows after sampling is what stops the population skewing toward the dataset's complete synthetic rows (ADR 0002). 2. Decode — ports/decoder.py, proven against a generated parquet, never a committed row (ADR 0016). Codes are 4-bit, indexed from zero, two to a byte, low nibble first (code_base: 0); the null bitmap is the sole authority on presence — a set bit means missing, LSB-first — and an absent bitmap means fully populated, which is how the 400,000 synthetic rows survive at all; attribute_overrides supersede the code unconditionally, are out of vocabulary without exception, and are folded: structured missingness (null, Not applicable) → absent, a real value the enum cannot express (65+) → absent, never admitted raw or guessed into a band, because the gates and filters rest on a closed vocabulary and a demographic is never synthesized — and counted as unexpressible in coverage and the preview, since the loss follows the value. The vocabulary is consulted before the missingness sentinels. The decoder checks its own output against the corpus's populated_attribute_count — equal to the bitmap present-count, not the non-zero-code count — and fails loudly on a mismatch, naming the row. Eligibility reads the same decoded labels, so an override that folds a conditioning field to absent can never slip into a persona. 3. Sample — seeded np.random.Generator; quotas from the shares the brief's audiences declare (all-or-nothing, summing to one); a brief declaring no audiences draws uniformly from the eligible pool, and its gates then judge the draw against the category's measured targets where the ontology carries them. admissible_sources — a recorded PopulationParameters field, on the manifest, never read from the environment — restricts which sources a study may draw from; a row outside them never enters the sample, and reaching a quota by admitting a weaker source is a choice the preview shows before the draw. A quota that cannot be filled climbs a fixed ladder, each rung recorded as a Relaxation on the gate report: widen an ordinal predicate by one band, then drop the least relevant non-conditioning filter using the ontology's relevance_order, then accept the shortfall so the achieved mix tells the truth, and fail only when an audience matches nothing (a preview reports that ladder's rungs without drawing a sample). The conditioning set is never a rung — trading it for coverage would undo ADR 0002. 4. Gate — a tagged union, never nullable twins: CategoricalGate on categorical marginals (χ², pass at p > 0.05) and OrdinalGate on attributes the ontology declares ordinal, using their band→midpoint scale (pass at ks_similarity ≥ 0.80, where ks_similarity = 1 − D — named for direction, because a threshold on a bare ks reads backwards). Gates run on measured or extracted attributes only (never a synthesized or calibrated value, which the gate would be judging an invention), and the verdict is the draw's, enforced before any model is called. Each distribution gate is judged against a reference it records: the study's own design — each audience's eligible pool weighted by its requested share — or, for a study declaring no audiences, the category's measured CategoryTargets on the ontology. A targeted study is never held to category targets, since it departs from its category by design. GateReport.reference and the new GateReport.evidence are both computed as the weakest claim the report supports: evidence is the weakest FieldOrigin among the gated attributes (carried per attribute in attribute_origins, never on a gate result — a result is a statistic, a tier is a property of what it was computed from), so a pass is never read as stronger evidence than its weakest gate rests on; reference is CATEGORY_TARGETS only when every gate was judged that way, and a gate may not claim CATEGORY_TARGETS on an attribute that is not MEASURED — "matches the measured category" cannot rest on a model reading product reviews. Each result carries its threshold and computes its verdict, and GateReport.overall is computed from those — all serialize, and a supplied value that contradicts the computation is refused. Any FAIL rejects the population. A Population aggregate joins personas, manifest, gate report, graph and communities with the BriefPack and enforces every cross-persona invariant: conditioning and field domains come from the ontology, never from the persona; gated attributes are measured or extracted for every carrier; the report's attribute tiers agree with what the personas carry; the source mix matches the personas — a population mixing sources takes the weakest tier present, so adding synthetic rows is visible in the verdict. The gate report carries the population's source_mix so any skew the conditioning filter induced is visible, and the graph's measured attribute assortativity so how much attributes shaped the structure travels with the population. 5. Project — the ontology's relevance order selects fields; the conditioning set lands in Persona.conditioning, everything else in Persona.attributes; sparse completion batched one call per ~25 personas of the same audience, and constrained to the corpus's own value set for that attribute — a choice among real values, never free text, since an invented value silently breaks the audience filters and gates that never saw it; every projected field states a FieldOrigin of MEASURED, EXTRACTED, SYNTHESIZED or CALIBRATED — the tier a corpus field carries comes from the source and the field's assignment_type together (ADR 0017), a completed field is SYNTHESIZED, and the tier is never inferred from an absent key. 6. Patch — PersonaPatchSource applied; default adapter is a no-op. 7. Embed — mean-pooled attribute-text vectors into one contiguous array; EmbeddingRef on each persona. 8. Graph — a ring lattice over a seeded permutation of the personas (clustering, as in Watts–Strogatz; the permutation keeps lattice neighbours from meaning shared audience) merged with a Barabási–Albert graph (hub tail), then a homophily rewiring pass: a homophily_strength share of ties is reconsidered, the higher-degree end is kept, and the tie moves to the most similar of 16 sampled candidates when that candidate is more similar than the current partner. Tie strength is attribute homophily only — categorical match and normalised ordinal distance between band midpoints, each attribute weighted 1/(rank+1) by the ontology's relevance_order, over the attributes both personas carry. There is no embedding term and no strong-tie flag, because homophily explains itself ("tied because both train three times a week") where a mean-pooled vector cannot, and leaving embeddings out removes an embedding call per persona from the common path. Homophily strength, ring degree, hub attachment and every graph, community and distribution threshold are tunable PopulationParameters with the engine's defaults, recorded on the manifest and never read from the environment — so a tuned graph or a loosened gate is visible wherever the population travels. Weights are quantised before becoming edges, since the graph hash covers them as floats. Seeded, bit-identical per seed. 9. Detect communities — Leiden over the weighted graph, seeded, across a γ search, selecting the qualifying partition with the highest modularity: modularity ≥ 0.4, 4–8 communities, each holding at least a fixed share of the population rather than a fixed count, so a small study is not structurally doomed. A population that yields no qualifying partition is still valid; polarization is then reported as not measurable, never as zero. Communities are discovered, so they appear only in outputs — never in an input file.

Graph gates: carried in the same GateReport as a third result kind, keyed by check rather than attribute — connectivity (share of personas in the largest component ≥ 0.98), clustering (average clustering coefficient ≥ 0.05) and degree shape (highest degree over mean degree ≥ 1.8, the hub tail) — and a failure rejects the population as a distribution gate does. Before them, a graph whose ties follow the declared audiences with assortativity at or above 0.9 is refused, since its communities would only restate the brief. Measured attribute assortativity is reported beside them, so how much attributes shaped the structure is a number rather than an assumption. "Zero isolates" is not a gate but an invariant of a valid population: an isolate gets one weak edge during generation, and Population refuses a graph that leaves any persona untied.

Salvage: MatrAIx persona_codes.schema.json (the decode contract), build_persona_1m_indexes.py, persona_1m_index.py, persona_1m_pool.py, calibration_targets.json. OASIS agent_graph.py for the networkx container only — topology is ours. leidenalg for community detection.

Perf gates: 10k-persona decode < 60 s cold, < 2 s warm. Postings filter must not open a shard. Full build() for n=2,000 under SyntheticCoresetSource < 5 s (this is the quickstart path).

Boundary tests (against FixtureCoresetSource + FakeInference, plus preview/interpret_audience against CoresetCatalog) — these replace the separate decode/sampling/projection unit tests entirely: - the packed decoder is proven against a generated parquet in the real format — an override that folds a conditioning field to absent, a row with no bitmap at all, a code at the vocabulary boundary, a populated_attribute_count that disagrees — never against a committed corpus row (ADR 0016); - an injected positivity skew in the fixture makes the gate FAIL and rejects the population; - no demographic or psychographic field is ever SYNTHESIZED — the contract-2 test, and it is only expressible here; - a gate judged against CATEGORY_TARGETS on an attribute that is not MEASURED is refused, and a report grades at the weakest tier among its gated attributes; - a preview reports 0 / 0 / n and 0 / m / n distinguishably, forecasts the ladder's rungs without drawing, and never reads a row — a catalog with no rows() proves it; - a filter naming an attribute no admissible source carries is refused at authoring with the coverage behind it; - a study restricting admissible_sources draws from those sources alone, and a population mixing sources grades to the weakest tier present; - a fixture row missing a conditioning-set attribute never reaches the candidate pool; - the gate report's source_mix reflects the conditioning filter's effect on source composition; - same (brief, n, population_seed) → identical population_hash, twice; - graph gates hold across 20 seeds; - filter widening on shortfall climbs the recorded ladder and is reflected in gate_report.relaxations; - a completion answering off-list leaves the field absent rather than inventing a value.

5.4 `inference` — every model call¶

Absorbs old M7 (router), M18 (cache). Reshaped by ADR 0021–0025: the engine speaks one protocol to a gateway the user runs, and owns every policy that changes what the trace must say.

Owns: all model access. No other module talks to a model endpoint, and no module imports a provider SDK or a gateway library (ADR 0021). Hides: the HTTP client, the batch queue and concurrency ceiling, request and token rate limiting, retries and backoff, the pinned fallback, served-model verification, cost recording and its source, the persistent cache and its sample keys, seeds, structured-output requests and repair, embedding batching, OpenTelemetry spans.

Interface: complete(requests) -> outcomes · chat(role, messages, *, temp, max_tokens, template_id) -> Completion | CallFailure (one request over complete) · embed(texts) -> EmbeddingResult — the vectors in input order, the normalisation applied, the dimension the run fixed on, and the cost event of every capped batch.

complete is synchronous to its caller and concurrent inside, and returns one outcome per request in request order — a Completion or a recorded failure, never an exception for one call and never an absence (ADR 0023). A Completion carries the text, template id, prompt hash, latency and the exact CostRecorded event it produced; the runner appends that event as-is, so cost has one source. Every cost event records its route — primary, fallback or cache — the pinned and the served model, and a cost_source of gateway, price_table (a declared price on reported usage), estimate (a declared price on estimated usage), cache (served from the cache, billing nothing) or unknown (ADR 0012).

Transport: OpenAI Chat Completions and Embeddings over httpx, to a base URL the user configures; the request bytes the engine hashes are the bytes it sends. The documented default gateway is a self-hosted LiteLLM proxy, which reaches Bedrock, Anthropic and hosted providers; vLLM and Ollama are reached directly. Gateway-side retries, fallbacks and caching are documented off, because a substitution the engine cannot see cannot be recorded; the engine detects a gateway that ignored this where it can — a served model outside the pin's accepted identifiers is a recorded pin failure on every call, and a response that reports no usage is billed from the declared price or recorded as an unknown cost, never guessed.

Internals: 1. Resolve role→model from ModelPins, failing before any network call if unpinned. A pin declares the served identifiers it accepts, its structured-output capability, whether it honours seed, and optionally its price. 2. Queue — a bounded queue so a tick of 620,000 turns never becomes 620,000 coroutines; a concurrency ceiling set to the endpoint's real capacity; an adaptive limiter on requests and tokens per minute, estimating tokens from text length and correcting from reported usage, lowering concurrency on sustained 429s and restoring it on success. One async loop on a dedicated thread, so the synchronous facade works from inside a running loop. 3. Cache — persistent, in the user cache directory, keyed by served model, template id and hash, exact request bytes and — for any call above temperature zero — a sample key from the replicate seed, persona and tick, so replicates never share a sample (ADR 0025). Replay reads the trace, never the cache. 4. Dispatch and retry — 429, 5xx, timeouts and connection errors retry with backoff honouring Retry-After; 400, 401, 403, 404 and 422 are fatal at once, and a small run of identical refusals trips a circuit breaker that fails the batch with one clear error — server errors are retried and never open it, since a client that outlived an outage must work once the endpoint recovers. Only a call that exhausted its retries on a retryable failure falls to the role's pinned fallback: a refusal, a pin failure or invalid output is not cured by another model, and handing over only the hard-to-parse answers would bias a study toward the fallback. An unpinned fallback is never used, and the embedding role has none. A call estimated above the token limit per minute can never be admitted and fails at once as exceeds_rate_limit. A failure records the status and the error's identifier-like type and code, never an error body, which can quote the request. 5. Verify — the first success fixes the served model for the run; a later call served by a model outside the pin's declared aliases, or by a different alias than the first, is a pin failure, a recorded failed call, and a cached answer from another alias is a miss. 6. Cost — the gateway's reported cost, else the pin's declared price (recorded as estimate when the response reported no usage), else unknown; budget enforcement refuses unknown unless unbudgeted spend is explicitly accepted. 7. Structure — a pin declaring structured output receives a strict response_format schema; every response is still parsed leniently (coerce_json), validated against the declared schema, repaired once with a stricter prompt, and otherwise recorded as an invalid-output failure — and distributions are validated against the corpus vocabulary where they are used, which is the caller that knows what it asked. 8. Seed — a seed derived from world seed, persona, tick and sequence is sent where the pin honours it, and recorded; provider determinism is never promised, because reproducibility comes from replay (ADR 0011). 9. Telemetry — a span per batch and per call, retries and fallbacks as span events, gen_ai.* attributes named in one module, metadata only unless content capture is explicitly enabled (ADR 0022). The core depends on opentelemetry-api alone; the SDK and OTLP exporter are the optional jahan[otel] extra.

Completion is sampled, not chosen (ADR 0024). Projection asks the model for a probability distribution over an attribute's vocabulary for each persona, and the engine samples the value with the population's seeded stream at a recorded completion_temperature. A distribution that does not cover the vocabulary or sum to one within tolerance is refused like an off-list value.

Configuration: the environment configures how a run executes, never what it measured. Endpoint URL, API key, concurrency, rate limits, timeouts and retry counts are execution configuration and unhashed; model pins, accepted served identifiers, capabilities, prices, temperatures and templates live on RunConfig and are hashed.

Roles: tier_a (bulk persona ticks — Persona-8B served by vLLM or through a gateway, small-instruct fallback), tier_b (first impressions, conversations, reflections, purchases — a frontier model), embed (one pinned model for anchors, responses and recsys alike — mixing embedding models invalidates SSR geometry; every vector's dimension is checked against the first), safety (a routed role; its moderation behaviour belongs to the caller).

Holdout evaluation: python -m jahan.holdout hides measured attitudes on Stack Overflow rows, projects them through the production path — in one of two arms, demographics (only the conditioning set, ADR 0019's question) or all (every other declared attribute), with the baseline always seeing exactly what projection saw — the same completion machinery, reached through an evaluation-only affordance, since no study may synthesize a psychographic but the technique can only be judged by running it — and scores the stated distributions with proper scoring rules — log loss and Brier score — against a smoothed demographic-conditional baseline built on the pool alone, beside marginal distance, calibration and recovered demographic dependence corrected for what sparse cells show by chance. The proper scores lead because the others cannot rank a projection on their own: marginal distance cannot see demographics, and calibration scores uniform guessing as perfect. The report records the pins, served models, seeds, completion temperature and row counts behind every number. It runs on the fake in CI and on a real endpoint when pointed at one — the engine's central number, owed by ADR 0019.

Salvage: MatrAIx openai_client.py (coerce_json; timeouts became httpx configuration), llm_usage.py's token accounting and cost_source provenance (without its LiteLLM price dependency). OASIS's async fan-out behind one concurrency ceiling, as a pattern only. Withdrawn: MatrAIx's per-provider branches in model_client.py (ADR 0021), its persona_model.py pin precedence (pins are recorded study inputs, never environment variables — ADR 0009), and OASIS's CAMEL-based agent model access, whose random model scheduling, uncontrollable memory and swallowed errors conflict with ADR 0009, 0011 and 0023.

Out of scope: the cost-and-time forecast (inference supplies token estimates and measured throughput; runner owns the budget and the forecast), streaming, the Anthropic protocol, the safety role's moderation behaviour, and any agent or tool-calling logic. Deferred and owed to runner: refusing unknown costs unless unbudgeted spend is explicitly accepted (PRD M4 user story 15) — inference records the source of every cost truthfully; enforcement belongs where the budget is.

Boundary tests: a scripted HTTP transport plays 429 sequences, timeouts, served-model changes, malformed JSON and reported costs through the real retry, limiter, fallback, cache and circuit-breaker code; outcomes return in request order whatever order responses arrive in; one failed call leaves the rest of a batch intact and recorded; an unpinned role raises before any network call; a served model outside the pin is a pin failure; two replicates never share a cached sample while a re-run of the same seed hits the cache; complete works from inside a running event loop; no telemetry span carries prompt content unless enabled; FakeChat is deterministic across processes for the same prompt hash; cache hit rate ≥ 30% on a baseline re-run of the same seeds.

5.5 `elicitation` — SSR¶

Old M9. Deliberately kept separate despite having one production caller. Reshaped by ADR 0026–0028.

Owns: converting free text to a Likert-5 probability mass, and the question that elicits the text. Hides: anchor sets and their embeddings, per-set similarities and scoring, averaging, ε and temperature, numeric-answer detection, the anchor checks.

Interface: batch-first, as ADR 0023 — score(responses, construct) -> outcomes, one per response in request order, each an SsrResult or a recorded elicitation failure.

Why it stays its own module despite agent being its only runtime caller: it has independent entrypoints (the anchor check and the mapping validation), its own acceptance gates, and it carries the engine's central scientific claim. Independent addressability is worth the shallowness; this is the one deliberate exception to P1.

The computation is the paper's (ADR 0026), ported with attribution from the authors' compute.py: similarity γ = (1 + cosine) / 2; per anchor set, subtract the least similar anchor's similarity and normalise, p_i = (γ_i − γ_min + ε·[i is the least similar]) / (Σγ − 5·γ_min + ε); average across sets; then apply temperature once, p^(1/T). ε and T are study parameters on RunConfig, per construct, defaulting to the paper's 0 and 1, and never tuned on validation data.

Anchors (ADR 0027): one domain-independent family per construct — purchase intent now, plus a satisfaction set used only for validation — of six hand-written sets of five statements with deliberately varied wording. Stored as immutable versioned JSON under anchors/<construct>/<version>.json, identified by content hash and pinned by RunConfig.anchor_set_hashes; a changed statement is a new version. Anchors are embedded once per run and cached by anchor hash; a second embedding model asking for the same version is refused, since similarities across spaces are not comparable. Before a version may be pinned, the anchor check must pass against the real embedding model: a frozen ladder of graded responses scores in increasing order, Spearman rank stability across sets exceeds 0.8, and varied responses do not collapse.

The question (Q7): elicitation owns a versioned, hashed question template per construct, based on the paper's "How likely are you to purchase the product?" with an instruction to answer briefly in the persona's own words, without numbers or ratings; agent includes it verbatim. A response carrying a rating-like number is a recorded elicitation failure with no distribution — no code path scores a model-emitted rating.

Two rules that are not negotiable: - Numeric elicitation is forbidden, and enforced by detection rather than by instruction alone. - Anchors and responses share one embedding model, pinned with no fallback (ADR 0012). SsrResult carries embed_model_id so a mismatch is detectable from the trace alone.

Interface out: SsrResult {response_text, per_set_similarities, per_set_pmfs, pmf, construct_id, category, anchor_set_id, anchor_version, embed_model_id, temperature, epsilon}, where pmf is computed as temperature applied to the mean of per_set_pmfs, so it cannot disagree with them, and the raw similarities make a change of ε or T arithmetic on the trace.

Embedding model (ADR 0028): Amazon Titan Text Embeddings v2, through a local LiteLLM proxy that also serves the chat models — neither of Bedrock's OpenAI-compatible endpoints serves embeddings, and OpenAI's embedding models are not on AWS. The paper used OpenAI text-embedding-3-small; the mapping validation measures whether the substitution holds, with Cohere Embed v4 next if it does not.

Validation (ADR 0028): the mapping claim — SSR recovers the rating a person gave from what they wrote — is validated on public human product reviews, about 500 balanced across stars, scored with log loss, Brier score and rank correlation against a baseline that ignores the text, using the frozen satisfaction anchors. The simulation claim — simulated purchase intent matches real people's — is owed; the trust level stays UNCALIBRATED (ADR 0008) until a purchase-intent benchmark exists.

Salvage: the SSR computation from pymc-labs/semantic-similarity-rating (Apache-2.0), ported with attribution and checked against its tests. MatrAIx json_survey.py adapted for the free-text mode.

Out of scope: a purchase-intent human benchmark, constructs beyond purchase intent, prompt-wording experiments, tuning ε or temperature, and assembling the persona's turn, which belongs to agent.

Boundary tests: the port reproduces the reference implementation's known answers; the ladder, rank-stability and non-collapse checks catch deliberately broken anchors; a response with a rating-like number is a failure, never a distribution; an embedding-model mismatch between anchors and response raises; outcomes return in request order and a failed embedding chunk fails only its own responses; changing ε or temperature on recorded similarities reproduces a fresh scoring exactly; raising temperature widens the distribution monotonically.

5.6 `agent` — the persona turn¶

Absorbs old M8 (agent), M10 (memory), and the previously unowned ContextBuilder.

Owns: one persona's reaction to one impression, and the persona-conditioning invariant. Hides: context assembly and token budgeting, persona rendering from ontology-selected attributes, memory retrieval scoring, reflection triggering, belief updates, tier routing, purchase-intent scoring, output parsing, guardrail enforcement, the character probe.

Interface: turns(jobs, *, chat, config, ontology, blocks, embed, stimulus_texts) -> tuple[TurnOutcome, ...]

A job is one persona, its PersonaState and its presentation; an outcome is a completed turn or a recorded turn failure, one per job in request order (ADR 0031). A single turn is turns([job])[0], and batching coalesces calls, never contexts — every persona still gets its own prompt. The agent stores nothing: beliefs, the persona's own memories and its reflection counters travel in and out, the trace is the store, and the runner is its only writer, so any worker can take any persona (ADR 0030). The rendered persona block travels beside the call in a PersonaBlockCache the runner holds: rendered once per persona per run, reused across ticks, its hash recorded on every turn. A turn that cannot be scored for purchase intent, because no anchor version is pinned, keeps its verbatim and records the elicitation failure in place of a distribution (ADR 0032).

The View is the public context of exactly the stimuli in the impression: like, repost, reply, upvote and downvote counts, reply ancestry, and the persona's tie strength and shared community with each author. It never carries another persona's attributes, beliefs or private reactions, nor any aggregate outcome — a persona that could see running adoption would react to the result being measured. Counts include only engagement from earlier ticks. The impression and view travel together as a presentation, and the turn records the view it was given (ADR 0010).

A persona reacts once per channel per tick, to everything it saw — an Impression holding one or more Exposures. Seeing two things side by side is not the same as seeing each alone, and social proof, the mechanic that distinguishes the feed from the survey room, only operates through juxtaposition. A survey-room impression holds exactly one exposure, so both environments share one code path. The reaction records the impression it saw and the subject_stimulus_id it is about, since intent is directed at a specific proposition even when several things were on screen (ADR 0003).

Why these merged: memory had exactly one caller and an interface as complex as its implementation. More importantly, ContextBuilder was referenced by the old spec but assigned to no module — and it is where persona conditioning lives. Conditioning is the load-bearing variable in the SSR literature: unconditioned personas produce optimistic, narrow distributions and rank correlation falls to roughly half. An invariant that decides the engine's validity cannot be co-owned. It now has exactly one owner, and one place to test.

Internals, one batched round each: 1. Tier routing — {first_seen, conversation, reflection, purchase, claim_audit} → tier B; everything else → tier A. The routing table is AgentConfig, data a study changes, not code. 2. Context assembly — the cached persona block, current beliefs read directly (no retrieval cost), retrieved memories, the impression and its view, then the frozen question (the free-text elicitation template on purchase tasks). Token-budgeted per tier on the wire format: memories drop lowest-ranked first, then beliefs, and a budget that would drop the persona block fails the turn instead. 3. Conditioning assertion — the block is checked for non-emptiness before dispatch. A turn cannot proceed unconditioned; the failure (unconditioned) is loud and recorded, and no model call is made. 4. Memory retrieval — score = exp(-Δt/τ_r) · importance · cos(emb_event, emb_stimulus) over the persona's own memories only, top-k per tier (A: 3, B: 8), τ_r = horizon/4. Memories are embedded once when written through EmbedPort and never re-embedded at retrieval. Importance follows a rule from the action taken and the size of the belief change — free and deterministic on tier A; on tier B the model may rate it inside the call already being made. 5. Dispatch via ChatPort, one batched complete per round: reactions, then stricter retries, then tier-B reflections, then tier-A probes. 6. Parse per task type, including optional belief deltas across the closed BeliefDim set and per claim; purchase-intent text is scored through elicitation when a passing anchor version is pinned, else kept beside its recorded failure. No prompt asks a model for a number. 7. Guardrails at parse time — the only two rules: a stimulus referenced that was in neither the impression, the view nor the persona's retrieved memories, and output that cannot be parsed. Either is rejected and retried once with a stricter instruction naming the allowed stimuli. A turn accepted on retry records the prompt hash it rejected; a turn whose retry also fails is recorded as a guardrail_violation carrying the impression, view, both prompt hashes and the rule broken, in place of a reaction. Never silently kept. 8. Belief update and reflection — deltas applied across dimensions and per claim, so a single-claim flip is visible when aggregate belief barely moves. Reflection fires on the per-persona cadence jittered from the run seed (4–8 ticks) or on max |Δ| > 0.3 across dimensions and claims together, as a tier-B call producing a further change plus one to three consolidated memories of high importance; a failed reflection consolidates by rule. Memories are capped with the highest-importance items kept, so state stays bounded over a long horizon. 9. Character probe — a seeded share (2%) of activated personas every 10 ticks is asked two questions whose answers sit in their own attributes, on tier A. Results record persona, questions, answers and agreement; the run's disagreement rate derives from them. A disagreeing probe never fails the turn or alters the reaction.

State crosses the boundary in PersonaState and returns in CompletedTurn (reaction, new memories, combined belief change, probe result, recalled memory ids, hashes); the runner applies it with advance_state and writes the trace records — turn, memories, belief snapshot, reflection, probe — from which rebuild_state reproduces the carried state exactly. Every seeded draw (reflection cadence, probe sampling and questions, identifiers) derives from the run seed and a named purpose, so two processes agree bit-identically.

Salvage: OASIS agent.py, agent_action.py, agent_environment.py (agent↔action↔env indirection; extend ActionType with buy, ask_peer, reject, complain). MatrAIx templating.py — the 1,290-attribute → prompt-section renderer, which is the hardest prompt-engineering problem in the salvage set and is already solved upstream. MatrAIx user_sim.py for conversational turns. Generative Agents (Park et al.) for the memory-and-reflection method — paper only, no code.

Boundary tests (FakeChat + FakeEmbed/DictEmbed + staged passing anchors; the suite never reaches a network): - the conditioning invariant — the persona block reaches the prompt and distinguishes it (four personas, four different blocks), and unconditioned dispatch is refused outright with no model call. That conditioning changes answers the way the literature reports is a real-model measurement, not a boundary test: a stand-in's answers are whatever the stand-in was written to return, so a fake asserting the effect asserts only itself. Owed as an evaluation beside the holdout (plans/m6-agent.md); - the block renders from ontology-selected attributes, once per persona per run through the runner-held cache, its hash on every turn; - a stimulus absent from context triggers the guardrail path exactly once (one batched retry round), then records a violation with two distinct prompt hashes; unparseable output takes the same path; the two rules are asserted as the only ones over the package; - reflection fires at the jittered cadence and at the belief-delta threshold — including a flip on a single claim — and not otherwise; the jitter is deterministic under a fixed seed; consolidation writes one to three high-importance memories and the cap keeps state bounded over forty ticks; - memory importance follows the rule with no model call on tier A; retrieval returns the k most relevant items under a controlled-geometry fixture (3 on tier A, 8 on tier B), never another persona's memory; memories embed once at write and never at retrieval; - tier routing matches the configuration table for every task and the table is overridable data; a pinned purchase turn records an SsrResult, an unpinned one the verbatim and its failure; no prompt constant requests a number; tier-B importance rides the same call and tier-A turns coalesce into one batch; - the probe samples its share on its cadence from the run seed, asks the persona's own attributes on tier A, records agreement, and never alters the reaction; the run's disagreement rate derives from the recorded results; - a survey-room impression carries exactly one exposure and produces exactly one reaction; - state rebuilt from replayed trace events equals the carried state; two processes agree bit-identically under a fixed seed; a resumed run continues from checkpointed state without re-running completed turns.

5.7 `world` — environments¶

Absorbs old M12 (env), M13 (platforms), M14 (recsys), M15 (schedule).

Owns: environment mechanics, platform state, who sees what, when agents act. Hides: SQLite platform state, per-platform action handling, four recsys modes, the activation clock, intervention scheduling.

Interface: reset(header) -> WorldDelta · step(tick, turns) -> WorldDelta

A step takes the previous tick's recorded Turns and returns a WorldDelta built from the trace's own record types — stimuli published, interventions applied, exposures dropped with the persona each was dropped for — plus one presentation (impression and view) per activated persona and channel. The opening delta is tick zero. The world never assigns event ids or sequence numbers; the runner is the partition's only writer. No world state crosses the boundary: a world resumes by reset and replaying its recorded turns through step up to the last closed tick, and replay must reproduce the recorded events exactly — which doubles as the determinism check. Internal checkpoints may speed replay up, but are not a contract (ADR 0011).

Time is declared, not assumed. A scenario states its tick_unit (hour, day, week) and horizon_ticks; interventions are expressed in ticks against them, and the unit travels forward into every digest so a report can label an axis truthfully. There is no second time vocabulary — the straggler case is simply the lowest tick among live worlds.

Internal structure mirrors upstream OASIS — env.py, platform.py, recsys.py, clock.py keep their upstream file shapes and license headers so quarterly upstream diffs stay mechanical (P6). The merge is at the interface: the rest of the engine sees one World port, not four modules. This is the deliberate compromise — one narrow interface outward, upstream-diffable structure inward.

Platforms:

Platform	Mechanics	Actions
SurveyRoom	isolated; every persona receives the stimulus, no social signals. The baseline environment	answer (SSR always applied)
SocialFeed	broadcast posts, comments, likes, reposts, quotes; exposure via recsys; visible social-proof counters	post, comment, like, repost, quote, follow
Forum	one class, two presets — `reddit_global` (any agent, any thread, hot-score ranking) and `community_scoped` (threads scoped to Leiden communities, recency + agreement ranking, no hot-score). Studies run `reddit_global`; `community_scoped` is a `WorldConfig` setting no study input selects yet	create_post, reply, vote
WOM	a graph channel, not a platform. After a reaction, `wants_to_talk(reaction, peer)` gates on sentiment strength and tie strength; delivery creates a next-tick exposure with `reason=wom`	send (implicit)

The two forum presets are mechanically similar and dynamically opposite — hot-score produces herding, consensus ranking produces slow hardening. Having both in one class makes that a study variable rather than a fork.

Recsys, all four OASIS modes retained: reddit_hot (hot-score math copied verbatim — do not "improve" it, its value is fidelity to upstream), twitter (OASIS's X refresh: in-network posts first, most liked first, then cosine of profile and post vectors × log((271.8 − age) / 100), using vectors from a separately pinned recsys_embed model), twhin (graph-aware, using generated degree centralities), random (the control arm — required to separate filter-driven effects from organic ones). A study's feed runs twitter when the run pins recsys_embed and random otherwise; its forum runs reddit_hot; twhin is a WorldConfig setting no study input selects yet. Exposure budget defaults to 3 stimuli per agent per tick; drops are logged with a reason. Exposures keep their per-stimulus attention, reason and seen flag, and are grouped into one Impression before reaching agent — grouped, not flattened.

Schedule: per-agent activation probability from involvement × daily rhythm, seeded. Interventions compose rather than overwrite — a promo during a launch is both, not the later one.

Salvage: OASIS env.py/env_action.py/make.py (PettingZoo-style loop; strip CAMEL, inject our ChatPort), platform.py + database.py + channel.py (already trace-shaped — extend, don't rewrite; add provenance columns at write time), recsys.py + process_recsys_posts.py (swap embedder to our port), clock.py (add straggler-aware semantics — variant worlds finish at different ticks), typing.py (ActionType, RecsysType).

Boundary tests: step() is deterministic under a fixed seed, bit-identical across two processes; random recsys mode produces measurably flatter exposure concentration than reddit_hot on the same fixture; exposure budget is never exceeded and an impression never exceeds it; a WOM delivery appears as a next-tick exposure whose view records the correct tie strength; replaying recorded turns from reset reproduces every recorded delta; composed interventions apply additively.

5.8 `runner` — execution and budget¶

Absorbs old M16 (scheduler), M17 (costs), M20 (sweep).

Owns: running a study to completion inside a budget, including enforcing degradation. Hides: job expansion, the worker pool, per-tick checkpointing, crash resume, the cost ledger, the degrade ladder and its application, sweep expansion, finalization to Parquet.

Interface: run(config: RunConfig) -> RunResult

A RunResult carries the registry entry and one outcome per world id — completed, partial or not_started, the last tick closed, and the degradation rungs applied — and the run's status is computed from its worlds. It holds no digests; analysis derives those from the partitions.

Why budget moved here. The previous design had the governor decide allow | degrade(level) | halt and then required three separate modules — agent (freeze tier B), world (subsample activation), scheduler (pause) — to each interpret that decision correctly. Three enforcement points for one policy means no single place can test "does the budget actually stop spending?", and a misinterpretation silently changes simulation fidelity mid-run, corrupting a study without failing it. The runner now applies degradation itself, by handing each tick a frozen TickPlan (activation rate, tier-B freeze, rung in force): it lowers the activation rate in the plan it hands to world (applied through WorldConfig.activation_rate), and it flips the tier-routing table it hands to agent. Those two modules stay unaware that budgets exist. Worlds run in parallel, one worker per world (one process each in production; threads sharing the trace port in boundary tests, same seam).

Degrade ladder (P3: one owner, one place — thresholds and the subsample are LadderConfig, configuration a study changes, not code; defaults 80% / 95% / 100% / pause above 150%, subsample to 0.40 from a base of 1.0): 1. 80% of budget → warn. 2. 95% → freeze optional tier-B work (reflections, optional audits). 3. 100% → activation subsample (0.62 → 0.40 is one study's setting of the rate). 4. Above 150% (pause_at) → pause the run. A world also pauses before a survey wave it cannot afford, since a wave is never thinned. Completed worlds are kept and labeled partial — partial results are valid results, and discarding them is the wrong failure mode.

Sweep is not a separate concept — it is run() over N scenario configs × seeds, with a worker pool and a shared budget. A sweep is a SweepPlan expanded into a RunConfig and run like any other, so it returns a RunResult; there is no separate sweep result, and per-cell digests for the heatmap come from analysis. The pattern comes from ASAL's main_sweep_gol.py (brute-force discrete sweep); nothing is copy-pasted, since JAX/CLIP is the wrong modality.

Determinism and resume: a scenario carries no seed — it describes conditions, not a draw. Replicate seeds live on RunConfig.seeds, and each cell derives world_seed = h(replicate_seed, variant_id) and world_id = sha256(canonical_hash(scenario), replicate_seed, population_hash)[:12] (ADR 0005, amending 0001) — identity covers the whole scenario, so a price sweep over one concept yields one world per price point, while the shared seed gives those price points common random draws. Expanding scenarios × seeds therefore produces genuinely distinct worlds, which is what makes the per-seed spread real and the 2σ anomaly thresholds meaningful. Resume is idempotent and "have I already run this sweep cell?" is answerable without a registry query. Each fully recorded tick is marked by a tick_closed event. Resume discards anything after the last closed tick, resets the world and replays its recorded turns (ADR 0011). Every rung the degrade ladder applies is recorded as a degraded event in each live world's partition at the tick it takes effect, carrying the activation rate and tier-B freeze now in force, so replay reproduces it and a degraded world is never compared silently with one that ran in full.

Salvage: ASAL main_sweep_gol.py (sweep pattern), rollout.py (run-then-score shape). MatrAIx llm_usage.py, budget.py, scoring.py (accounting plumbing), jobs.py/job_aggregation.py (run aggregation shape).

Boundary tests (in-memory adapters, synthetic cost stream) — these are the tests that could not be written before: - feeding a cost stream past each threshold produces the observable effect: tier-B calls stop, activation rate actually drops, the run pauses; - a paused run retains every completed world, labeled partial; - kill a run mid-tick and resume by replay — the result is identical to an uninterrupted run; a degraded world's partition records every rung applied; - the same sweep grid run twice produces identical world_ids and identical digests; - a brief_hash change refuses resume rather than silently continuing.

5.9 `trace` — the audit spine¶

Old M19.

Owns: the append-only record of everything that happened, and the typed read views over it. Hides: the SQLite→Parquet lifecycle, partitioning, the run registry, query shapes.

Interface: write(events: Iterable[TraceEvent]) -> None · view(run_id) -> TraceView · finalize(world_id) · registry.record(entry)

TraceSink, TraceView and RunRegistry are protocols in ports; TraceStore and SqliteRunRegistry in jahan/trace/ satisfy them, with in-memory fakes beside them for tests that need no filesystem. Every boundary test runs against a temporary directory.

TraceView is a typed, closed set of read shapes — events(filter), beliefs(persona_id), edges(), verbatims(grouping), resolve(trace_ids) — not an open handle to Parquet files. The previous design passed "trace views" as an undefined wide interface, which is what let derivation logic leak into two consumers.

Layout:

trace/
├── registry.db          run_id → config, hashes, seeds, pins, engine version, cost, status
└── runs/{run_id}/{world_id}/
    ├── live.sqlite      the tick-atomic record while the run is live or paused
    ├── events.parquet   sorted by (persona_id, tick), one struct column per payload kind
    ├── beliefs.parquet  one row per persona and tick, written by the view's derivation
    ├── edges.parquet    (u, v, channel, count, last_tick), written by the view's derivation
    └── partition.json   the header, contract version and finalized event count

The world's own state.db may sit beside a partition for convenience and is never read here. The exports/{study_id}/ client deliverable is a delivery format, not part of the spine, and belongs with report/cli; deferred deliberately.

Sizing: ~2k agents × 30 ticks ≈ 500k events per world ≈ 100–200 MB Parquet; verbose tier-B verbatims are ~7% of events.

Context is never stored whole — only its parts plus prompt_hash. Full prompts are re-derivable from the registry's pins and template versions. There is deliberately no field to put a full prompt in.

Write path: TraceEvent.model_construct on the hot path, batched; full validation runs in fake-mode CI. Payloads are a discriminated union over thirteen kinds — stimulus published, exposure dropped, turn, guardrail violation, reflection, memory, belief snapshot, probe, cost, intervention, degraded, tick closed, lifecycle — each the only record of what it describes; a turn carries the view its persona was given, verified against the partition's own earlier events (ADR 0010): a turn event carries the whole impression and its reaction, so what a persona saw side by side is never reassembled from separate events (ADR 0006). The union is what lets the Parquet writer fan out into typed columns instead of a JSON blob. A TracePartition is validated as a whole against a header carrying the run configuration, brief pack, population manifest, scenario and replicate seed. The header verifies every pin the run states against the object it pins and derives the world id (ADR 0009); the events must then be gapless, publish stimuli before showing them, name only the brief's claims, belong to personas in the population, use only pinned templates, anchor sets, embedding model and billed models, recall only earlier turns or reflections of the same persona, and move through a valid lifecycle. Events carry no seed and no contract version — world_id already resolves the world, the registry holds every seed, and the contract version is a constant across a partition, so storing either on 500k rows is pure redundancy on the hottest path in the system.

Read path is version-aware. A partition from a newer contract is refused; an older one is migrated forward through every registered migration newer than its contract, then validated strictly. The contract version is written once into the Parquet partition metadata and the run registry entry. write is strict; read_event dispatches on the partition's version and falls back to a permissive shape for older runs. Decide this now — a trace you cannot read under a newer schema makes replay a lie, and retrofitting it after runs are stored is painful.

Salvage: OASIS database.py and channel.py — already trace-shaped.

Boundary tests: 500k events round-trip through write and read with identical ordering by (persona_id, tick, seq) (measured: half a million cost events finalize in ~30 s into ~5 MB of Parquet); every read shape answers identically before and after finalization on the same world; a partition written under contract 1.0 loads under a synthetic 1.1; an event whose payload kind disagrees with its declared type is refused; resolve(trace_ids) returns exactly the referenced events or raises; registry entries pin every hash needed for replay; a duplicate (world_id, seq) is refused while an identical re-send is a no-op.

5.10 `analysis` — deriving meaning¶

Absorbs old M20's digest builder and M27's finding authorship. New module, and the fix for a real duplication.

Owns: everything derived from a trace — digests, verbatim clustering, anomaly detection, finding authorship, and the trust guard. Hides: clustering, statistics, anomaly thresholds, evidence resolution.

Interface: digest(view, *, scenario, population, seed, pinned_embed_model=None) -> OutcomeDigest · spread(digests) -> ScenarioSummary · cluster_objections(view, *, embed, threshold, seed, pinned_embed_model=None) · findings(view, *, embed, threshold, seed, pinned_embed_model=None) -> tuple[Finding, ...] · detect_anomalies(view, *, digest, replicate_spread, thresholds=...) -> AnomalyReport · trust_statement(...) -> TrustStatement

Why this module exists. The previous design computed objection clusters in two places — the sweep's digest builder and the report builder — from the same verbatims, with no stated owner. One concept, two implementations, guaranteed to diverge. Worse, the trust guard ("refuse any claim without trace resolution") sat in the renderer, meaning it could only be tested by rendering. Derivation now has one owner, and the guard runs where findings are authored. Everything is derived and nothing is judged: every number recomputes from the trace, and the only model call in the package is embedding verbatims through the run's pinned embedding model.

Digest: one world, naming its scenario, replicate seed and derived world id (checked against the world the view read). adoption = share-weighted top-two-box purchase intent, computed from the audience shares the digest carries (ADR 0007) — and None with a recorded reason and the count of unscored turns when the run scored no intent (ADR 0038), never zero. audience_pmfs (declared slices, authorable and comparable) and community_pmfs (discovered clusters) both reported where measured; polarization = size-weighted JSD across community PMFs, the emergent measure, with audience-level divergence alongside it, each None where unmeasurable. Beside intent every digest carries what every run produces: the action mix, mean and absolute belief movement, word-of-mouth deliveries and reach, and the degradation rungs the world ran under. A ScenarioSummary gathers one scenario's worlds — one entry per seed carrying its digest — with the population standard deviation between them (zero for one seed, reported as such) and a mark where worlds ran at different rungs (ADR 0037, ADR 0039). Comparing digests whose tick_unit differs is refused, as are worlds from different scenarios.

Anomalies are rule-based, not model-judged: trailing-window mean belief movement beyond twice the replicate spread of belief movement → herding, which needs a spread above zero and reports as not measurable without one, since twice a spread of zero is cleared by any movement at all. The yardstick is the spread of the same quantity the rule reads — belief_move_mean, computed in one place and carried by every digest — never the spread of adoption, which is a probability share rather than a signed move and is unmeasured on every run until an anchor version passes; a split in the sign of belief moves past a threshold share → backlash; flop needs adoption, so until intent exists it reports as not measurable with its reason rather than as absent (ADR 0040). Thresholds are configuration with documented defaults, and the applied threshold travels beside the observed value on every anomaly. Deterministic, cheap, and testable — no judge model, no calibration burden.

Finding authorship is deterministic extraction, not generation: objection clusters from verbatim embeddings (cosine similarity at a recorded threshold, ties broken from a derived seed, each labelled by its medoid verbatim quoted as written — ADR 0041), belief-shift findings from recorded belief changes, WOM-path findings from the derived edges. Every finding is constructed with its evidence trace IDs resolved against the view at authorship, because Finding cannot be constructed without them, and each carries the disconfirming test that would show it wrong (ADR 0042).

Trust guard, enforced here: - Every finding resolves to trace IDs, or it is not created. - Every finding carries a disconfirming_test — the real-world check that would falsify it. - A digest or finding built from a trace in a different embedding space than the run's pin is refused. - No path in the package calls a chat model, asserted over the package sources. - Calibration status is stated once per run, not per finding — every finding from one run necessarily shares it, so repeating it invites someone to vary it. Finding carries only what genuinely varies per finding: kind, statement, evidence IDs, confidence, and a disconfirming test. - The run's TrustStatement is UNCALIBRATED; a higher level needs a calibration reference pinning its benchmark and human study by hash, with measured similarity and rank attainment at or above 0.80 (ADR 0008). CATEGORY_BENCHMARKED and PROSPECTIVELY_VALIDATED require a CalibrationRef, which nothing in this repository can produce — the benchmark harness is deferred (§12). The engine cannot overclaim by construction rather than by discipline.

Boundary tests: a fixture trace with a planted herding pattern produces exactly one herding anomaly at the right tick; a planted sign split produces backlash while a quiet trace produces neither; flop without adoption is unmeasured with its reason; identical verbatims cluster identically across two runs and paraphrases group on a built fixture, embedded exactly once; constructing a finding without evidence raises; a trust level above UNCALIBRATED without a calibration reference raises; every finding from a fixture trace resolves against that trace; the regenerated offline study in docs/evaluations/m10-analysis/ digests end to end with every figure recomputed from raw events.

5.11 `report` — rendering¶

Old M27's renderer half.

Owns: turning findings and digests into documents. Nothing else. Hides: templates, markdown/JSON parity.

Interface: render(findings, digests, pack: ReportPack) -> RenderedReport

Deliberately thin. All judgment lives in analysis; this module cannot introduce a claim, because it only formats Finding objects, and those arrive pre-validated. ReportPack carries everything else the document must state: the run configuration (pins, seeds, template and anchor-set hashes), the run's single TrustStatement, the brief pack the assumption ledger is gathered from, the anomalies and objection clusters to list, the recommended real-world validation, the engine commit, the engine versions the run was forced across, and what a forced resume was forced past. Rendering validates a Report from the pack, the findings and the digests, and renders both formats from it, so every rule that contract holds — one digest per world, each world one this run configures, no finding id twice, no anomaly past its scenario's horizon — is checked where the document is written. RenderedReport carries markdown (human) and JSON-ready data (the mockup pages consume the JSON — that shape is a real contract carrying the version it was rendered under), both produced from one intermediate over the same finding set, so they cannot diverge. Every report states the trust level once, discloses pins, seeds, template versions and the engine commit, marks a run forced across engine versions, and ends with the recommended real-world validation section; findings carry only their own confidence, so "this audience was small" is never confused with "this engine has never been benchmarked". Where a digest reports adoption, polarization or audience divergence as not measurable, the report says so where the number would have been, and it prints what every run produces whatever it scored — the turn count and how many went unscored, the action mix, belief movement and word-of-mouth reach, and the rungs a world ran degraded under (ADR 0038). Free text a model or a persona wrote — statements, disconfirming tests, cluster labels, assumptions — renders on one line, so a quoted verbatim holding a line break cannot become a heading of the document that quotes it; the record and the JSON keep it as written. A finding's evidence is elided on the page after the first few ids and carried whole in the JSON, because every turn of every persona can be evidence and a real study's finding cites thousands. Stable ordering everywhere — findings by kind then id, clusters by size then label then world, digests by scenario then seed — so two renders of one finding set are byte-identical and a diff means a difference in findings.

Boundary tests: markdown and JSON contain the same finding ids in the same order; the trust statement appears exactly once and the validation section last; the method disclosure names pins, seeds, template versions and the engine commit; a forced run says so; removing trust, validation or method refuses; every finding shows evidence ids, confidence and its disconfirming test; the ledger includes what the brief left unstated; unmeasured adoption is stated with its reason; clusters render their quoted label; shuffling inputs changes nothing; the package imports nothing from analysis and performs no arithmetic beyond string formatting, asserted over the module; every printed number appears in the digest, finding or pack field it came from. (CONTEXT.md already defines every rendered concept — Finding, Trust Level, Confidence, Disconfirming Test, Unmeasured, Assumption Ledger, Method Disclosure — so the glossary stands unchanged.)

5.12 `cli` — entrypoints¶

Entry is python -m jahan.cli. Plumbing only: the CLI wires modules together, formats errors and writes files. It derives nothing, renders nothing and decides nothing a study did not already state.

Command	Behavior
`coreset-gate --brief b.yaml --n 1500 --seed 4021`	gate report + population manifest, no world starts
`ssr-replica --anchors <set-id> --construct <construct> --anchor-version v1 --population manifest.json`	the anchor check (ladder, rank stability, non-collapse) with its diagnostics; a failing version is reported as failing and is not pinned
`concepts run brief.yaml [--fake]`	one baseline study: report.md + report.json + run_id
`sweep run --grid grid.yaml --budget 42`	the grid as one run under one budget, with per-scenario summaries carrying each cell's seeds and their spread
`ontology check --ontology o.json`	an ontology draft against the corpus codebook; a refusal names what the codebook does have that resembles it
`brief check --brief b.yaml`	a brief against the engine's contracts, printing the assumption ledger it assembles

Every command prints its run_id and the path it wrote to. Artefacts land in a run-named directory, never in the caller's working directory: report.md, report.json, digest.json, result.json (written last, so completed implies its siblings), manifest.json, personas.json, ontology.json, brief.yaml, gate-report.json, trace-summary.json, progress.json, per-scenario summary-NN.json, and the trace/ store beside them. Exit codes come from the exception class, mapped in one place (cli/_errors.py): SimError 1 (crash), GateFailure 2, BudgetExhausted 3, SchemaVersionError 5 — and an unmapped exception exits 1 rather than something arbitrary. A malformed command line is unmapped too: argparse ends one with its own exit code 2, which is this table's failed gate, so a usage error exits 1 with argparse's message and --help exits 0. A study's verdict is never mistaken for a defect.

--fake uses stub inference over a synthetic coreset derived from the study's own brief and ontology, requires no API key and no dataset download, and runs the full pipeline end to end — population, world, agent, runner, trace, analysis, report, with no module stubbed. This is the path a new user hits first, so it is exercised by the boundary suite rather than left as a manual step — including a study long enough for a persona to reflect, which is a third prompt shape the stub has to answer; two fake runs under one seed produce identical reports. (The repository carries no CI configuration yet, so "runs in CI" means "runs in the suite".) --coreset-fixture runs offline studies over committed test rows instead. A real study pins --model and --embed-model, reads its corpus from the user's shard cache (refusing with the fetch command when nothing is cached) and its endpoint configuration from the environment, passed through without reinterpretation.

A study that measured no adoption still exits 0 and reports why. An exhausted budget exits 3 after the completed worlds' artefacts are in place. Re-running with the same run id resumes: finished cells replay without stepping, unfinished cells continue. A resume whose inputs moved refuses by name; --force proceeds, and what it was forced past is recorded in the registry and stated in the report (ADR 0036). (CONTEXT.md already defines Study, World, Sweep, Rung and Unmeasured, and SALVAGE.md's cli rows — the sweep-grid pattern and pins-seeding-recorded-defaults — describe exactly this wiring, so both stand unchanged.)

Boundary tests: commands invoked in-process against temporary directories; the fake end-to-end run, the failing-gate and exhausted-budget exits, both pre-flights (including a passing anchor check through the real gate), the offline-provable slices of the real path, and sweep summaries, degraded marking, budget-kept partials and resume-without-recompute are all in the suite. The suite reaches no network.

5.13 `web` — the engine over HTTP¶

M13. The engine with a face on it, and the discipline that keeps the face honest (ADR 0043–0046).

Owns: serving the engine over HTTP, and the discipline that keeps it thin. The five shapes, analysis's digest, findings, clusters, anomalies, report, trace summary, trajectories and workspace summary — live, from a running or finished study, with the registry deciding which backend a view opens. Hides: serialisation, filtering, paging, the run lifecycle (studies as subprocesses, cancellation, resume, orphan sweep).

The rule that shapes everything: the web layer derives nothing (ADR 0045). It serialises, filters, pages and streams. Every number it returns is a field of something analysis, trace or a contract produced — the five shapes, TraceSummary, the digest, findings, clusters, anomalies, the report, the workspace summary. A panel wanting an underived number is blocked until analysis derives it. An AST boundary test asserts the package performs no arithmetic over what it serves — none of the calls report's discipline test forbids (sum, min, max, abs, round, pow), no statistics import, no binary operator outside type unions. What a running study has done so far (ticks closed, turns landed, the rung in force) is analysis.world_progress, and which runs have written a report is named to analysis.workspace_summary rather than counted here.

Routes follow the five shapes one to one — /api/runs/{run_id}/events, /api/runs/{run_id}/worlds/{world_id}/{beliefs|edges|verbatims|resolve} — with TraceSummary at /summary, analysis's digest, findings, clusters, anomalies and report as artefact reads, /api/codebook and /api/ontologies behind the ontology builder, /api/briefs/validate, /api/gate (the population gate on a brief, answering with its report even when the draw fails) and /api/status for intake, /api/trust (the ladder and the floors the schema enforces) and /api/anchors-checks for the trust page, /api/runs/{run_id}/resolve (evidence across every world of the run — a finding cites each replicate, and a world's own resolve refuses a sibling's ids), and run lifecycle at POST /api/runs, DELETE /api/runs/{run_id}, POST /api/runs/{run_id}/resume. Each of the five shapes takes a typed filter (EventFilter, VerbatimGrouping) — never free keyword arguments. No endpoint returns a filesystem path, a cursor or a frame — a refusal names the file that was wrong (brief.yaml), never where it lives. Every refusal is one shape, {detail: <sentence>}; a study's request is a closed model (StudyRequest) that turns away bad numbers, unknown keys and any attempt to name an endpoint or a key before a process is started, and a run id must be one the trace itself would accept. Reading never writes: opening a TraceStore creates what is missing, so a read of a run that never recorded a trace answers 404 rather than leaving a registry behind.

A live run and a finished run answer identically through the same endpoints: the registry decides which backend a view opens, and TraceStore.view hides which. A run started from the interface is a subprocess (web/_lifecycle.py): the registry is authoritative for what happened, the process table only for whether it is still running, a study outlives the server that started it (its own session; its pid in launch.json), so the sweep at server start adopts a survivor it can recognise — the pid must be alive and carry the run id in its own argv, so a recycled pid is never mistaken for it — and marks only a dead run partial; a study that died mid-session reads partial, not running; a study whose process is alive is running until result.json says otherwise; its output goes to launch.log, and one that died before recording anything says why. Cancelling loses at most the tick in flight because a tick is recorded whole or not at all. Resuming is a re-run with the same id. The runner publishes progress — status, recorded cost, ticks closed — to the registry entry as it works (the entry is the published cache of what the trace already says), so spend is something a person can act on rather than read afterwards. result.json is written last, so a run reporting completed has its report, digest and trace summary on disk. Execution configuration stays in the server's environment: base URL, key, concurrency and rate limits are never hashed, never rendered, never accepted in a browser — /api/status reports only whether an endpoint is configured.

One implementation, two delivery paths. A derived shape is computed once in Python: the CLI writes it into the run directory at the end of a run (trace-summary.json, personas.json, ontology.json, progress.json, and result.json last), and the API serves the same function live — TraceSummary is recomputed from views only when the stored file is absent. Neither path recomputes what the other computes.

One turn's prompt is reconstructed and verified, never stored (ADR 0046). agent.reconstruct_turn re-renders the persona block from the population's records, replays beliefs from snapshots and turns, resolves memories by id and stimuli by published text, then displays the messages only when their hash matches the turn's recorded hash — anything else says so and why. The web endpoint serves it transiently; nothing in the interface persists a prompt.

Trust is stated once. The run's TrustStatement appears once per study view and never on a finding; the trust page shows the ladder and what a level above UNCALIBRATED requires; nothing in the interface writes or overrides a trust level; a finding's own confidence renders beside it and is never presented as the engine's calibration.

Test posture: the API is exercised in-process against temporary directories with the fake backend — the suite reaches no network. A live run and a finished run answer identically through the same endpoints; the package performs no arithmetic over what it serves, asserted over the package in the shape report's discipline test uses. The server ships as the jahan[web] extra (fastapi, uvicorn); the core's ten runtime dependencies are unchanged.

5.14 `ui` — the Next.js interface¶

The only module not written in Python, and the only one that holds no engine logic. It renders what the API returns and configures what a study states: intake → ontology builder → population → run → trace → atlas → report → trust. The interface speaks the glossary — Population, never cohort; Audience and Community, never segment — with a redirect from the old cohort address.

One implementation, two delivery paths, same rule as the shapes: a derived shape is computed once in Python; the CLI writes it into the run directory, and the API serves the same function live. The interface's own API routes are proxies to the engine (JAHAN_WEB_URL, default http://127.0.0.1:8000 — the one thing the interface is configured with) and nothing else: it reads no run directory, starts no process, mints no run id and finds no checkout, asserted over its source. There is no second mode. Every engine number arrives over HTTP, and a refusal keeps the status and the sentence the engine gave (lib/refusal.ts), so a brief the engine turns away, or an attribute the corpus does not carry, is explained at the screen rather than reduced to a status code. The shell states only what the engine said: recent studies, its version, whether an endpoint is configured, the spend the workspace summary derived — a single-operator application has no plan, no quota and no account, and shows none.

Nothing generated is displayed as derived. Cluster labels are quoted verbatims; findings carry their evidence and disconfirming tests; a quantity that could not be measured renders its reason where the number would have been. The run's calibration states once per study view (TrustLine) and is never adjustable; a fake run is marked as fake wherever it appears; audiences and communities are presented as different things. Where a chart would be empty the diagnostic appears instead — no communities formed, unmeasured with its reason, a failed gate readable because a study that never ran is the case the population page most needs to explain.

The interface is exercised against the real artefacts of a recorded run, never against fixtures invented for it — three of this engine's worst defects lived behind fakes that answered whatever the test wanted. A test that reads the interface's source for a string cannot fail when the interface is broken, so the suite starts the engine API and the Next.js server (test_interface.py), asks the interface's own routes for what its pages ask, and has a browser render every page — a page that throws while rendering answers 200 to every request a route can be asked. The form's brief is written out by js-yaml under the YAML 1.1 schema PyYAML reads with, and every brief the engine holds is put through the form and back through the engine's own validation; an audience's filter is a value, a list or a range, and the form keeps which. Its pages are intake (brief, claims with their sources and evidence, audiences, assumption ledger, the asked task and anchor version, endpoint status without keys), ontology builder (codebook search with declared value sets, drafts refused as typed against the corpus, versioned saves), population (gate statistics with thresholds, requested/achieved mix, origins, communities), run (watched live: ticks closing, turns landing, spend against budget, rung in force, cancel and resume), trace (the five questions), persona history (one persona's timeline with prompt reconstruction), atlas (adoption plane, trajectories, replicate spread, ranking and risk findings), report and calibration.

6. Cross-cutting contracts¶

Four rules, each with exactly one enforcement owner (P3).

Contract	Owner	Enforcement
No unprovenanced numbers — every claim resolves to trace IDs	`analysis`	`Finding` cannot be constructed without evidence IDs; the guard runs at authorship
Measured ≠ extracted ≠ synthesized — a survey answer, a model's reading of corpus text and a model-completed field never mix silently	`population`	an explicit `FieldOrigin` tier on every projected field, derived from the source and its `assignment_type` (ADR 0017), never inferred from an absent key; demographics/psychographics are never completable; asserted in a boundary test
Replayability — run_id + seeds + pins + template hashes reproduce a run	`runner` + `trace`	`ModelPins` rejects unpinned IDs; `config_hash` covers schema version; registry stores every hash
Nothing auto-crowned — winners are hypotheses	`analysis`	`disconfirming_test` is a required non-empty field; every report ends in the next real-world test

7. Salvage inventory¶

Full per-file detail is in SALVAGE.md; this is the module-level mapping. Keep license headers intact in every forked file and credit CAMEL-AI, Sakana AI, and MatrAIx in NOTICE.md.

Source	License	Feeds	Verdict
OASIS `env.py`, `env_action.py`, `make.py`	Apache-2.0	`world`	COPY/ADAPT — strip CAMEL, inject `ChatPort`
OASIS `platform.py`, `database.py`, `channel.py`	Apache-2.0	`world`, `trace`	COPY — add provenance columns at write time
OASIS `recsys.py`, `process_recsys_posts.py`	Apache-2.0	`world`	ADAPT — hot-score math verbatim; swap embedder to our port
OASIS `clock.py`	Apache-2.0	`world`	COPY/ADAPT — add straggler-aware tick semantics
OASIS `agent.py`, `agent_action.py`, `agent_environment.py`	Apache-2.0	`agent`	COPY/ADAPT — extend `ActionType`
OASIS `agent_graph.py`	Apache-2.0	`population`	ADAPT — networkx container only; topology is ours
OASIS `generator/`	Apache-2.0	—	SKIP — grounded sampling replaces persona invention
ASAL `main_sweep_gol.py`, `rollout.py`	Apache-2.0	`runner`	ADAPT (pattern) — reimplemented in plain Python; JAX/CLIP is the wrong modality
ASAL `main_illuminate.py`, `main_opt.py`, `asal_metrics.py`	Apache-2.0	`analysis` (phase 4)	ADAPT (pattern) — see §12
ASAL JAX substrates, `clip.py`, `dino.py`	Apache-2.0	—	SKIP — wrong substrate and modality
MatrAIx `persona_codes.schema.json`, index builders, pool	MIT	`population`	COPY — the decode contract
MatrAIx Persona-1M parquet shards	`matraix-research-only` (ADR 0016)	`population` via `CoresetSource`/`CoresetCatalog`	Never bundled, never downloaded implicitly — the user fetches shards explicitly with `hf download` into a cache outside the repository and accepts the dataset's own terms; `HfCoresetSource` reads that cache and verifies each file against the manifest on use; `SyntheticCoresetSource` keeps the quickstart working with no corpus at all
MatrAIx `openai_client.py`, `persona_model.py`, `llm_usage.py`	MIT	`inference`	ADAPT — `coerce_json`, pin precedence, token accounting; `model_client.py`'s provider branches withdrawn (ADR 0021)
MatrAIx `templating.py`, `user_sim.py`, `json_survey.py`	MIT	`agent`, `elicitation`	COPY/ADAPT — the 1,290-attribute prompt renderer
MatrAIx `survey_task_content.py`, `survey_eval.py`	MIT	`world` (SurveyRoom)	ADAPT — strip eval framing, add SSR free-text mode
MatrAIx Harbor runtime, web app, browser/OS agents	MIT	—	SKIP — orthogonal
SSR — arXiv 2510.08338 and `pymc-labs/semantic-similarity-rating`	Apache-2.0	`elicitation`	ADAPT — port the computation from `compute.py` with attribution (ADR 0026); anchors are not published and are written here (ADR 0027)
Generative Agents (Park et al.)	method	`agent`	no code — memory/reflection recipe
Leiden (Traag et al.)	method + `leidenalg`	`population`	library

Salvage rules: one inference module — no module talks to a model endpoint directly, and none imports a provider SDK (ADR 0021). Forked OASIS code keeps upstream structure and headers (P6); pin upstream SHAs and review diffs quarterly for recsys/clock fixes. No persona-invention code from any source. Any salvage that changes golden-run output gets reviewed and re-committed deliberately, never silently.

8. Testing strategy¶

Three layers, with distinct jobs. The middle layer is the primary one and is where most test code should live.

Layer 1 — property and contract tests (schemas). Fast, exhaustive, hypothesis-driven. PMFs sum to 1; hashes are stable and order-independent; round-trips are lossless; the leaf rule holds. These guard data, not behavior.

Layer 2 — boundary tests (every deep module). The primary layer. Each module is exercised through its public interface with in-memory adapters, asserting observable outputs. This is the layer the previous architecture was missing, and the reason several invariants — no synthesized demographics, budget actually stops spending, conditioning changes distributions, planted anomalies are detected — were previously untestable at any granularity below a full pipeline run.

Layer 3 — golden runs (characterization). Seeded full-pipeline runs under FakeInference + SyntheticCoresetSource, snapshotting final PMFs, event counts, and cost. These detect change, not correctness — a consistently wrong decode passes forever — so they are a drift alarm, never a correctness argument. Any diff must be reviewed and re-committed deliberately.

What is explicitly not tested with unit tests: internal helpers of deep modules. Testing a private scoring formula couples tests to implementation and blocks the refactors the deep-module structure exists to enable. If a helper feels like it needs its own test, that is evidence it should be its own module with its own boundary.

CI: layers 1–3 all run with zero API keys and zero dataset downloads. Runs against real models are a separate, manual, logged benchmark class.

9. Repository layout¶

jahan/
├── README.md
├── CONTEXT.md                   the glossary
├── LICENSE.md                   FSL-1.1-ALv2
├── pyproject.toml
├── mkdocs.yml                   the documentation site
├── docs/                        architecture · salvage · ADRs · PRDs · evaluations
├── frontend/                    the interface
├── jahan/
│   ├── schemas/                 leaf: contracts only
│   ├── ports/                   port protocols + all adapters (fake, fixture, synthetic, http, hf)
│   ├── brief/
│   ├── population/              decode · sample · gate · project · embed · graph · communities
│   ├── inference/               client · batch queue · limiter · cache · fake
│   ├── elicitation/             SSR + anchor sets
│   ├── agent/                   context · conditioning · memory · turn
│   ├── world/                   env · platforms · recsys · clock  (upstream-shaped)
│   ├── runner/                  scheduler · budget · sweep · checkpoint
│   ├── trace/                   sink · views · registry
│   ├── analysis/                digests · clustering · anomalies · findings
│   ├── report/
│   └── cli/
├── anchors/                     versioned SSR anchor sets
├── ontologies/                  versioned category ontologies, `<category>/<version>.json`
└── tests/
    ├── property/                layer 1
    ├── boundary/                layer 2 — one directory per module
    ├── golden/                  layer 3 + committed snapshots
    └── fixtures/                mini coreset shard, stub anchors, fixture traces

ports/ holding both protocols and adapters is deliberate: adapters are infrastructure, and keeping them out of the core packages means no core module can accidentally import a concrete adapter.

Core dependencies: pydantic, pyyaml (brief intake), pyarrow, numpy, scipy (distribution gates: the chi-squared survival function), networkx, leidenalg with igraph (Leiden's graph, imported directly), httpx (the one OpenAI-compatible transport, ADR 0021), opentelemetry-api (spans, no-op until configured; SDK and OTLP exporter in the optional otel extra, ADR 0022). SQLite for live state and traces until scale demands otherwise.

10. Build phases¶

Each phase ends in something runnable.

Phase	Weeks	Modules	Exit artifact
0	1–2	`schemas`, `ports`, `population`, `inference`, `elicitation`, `cli` (2 cmds)	`coreset-gate` produces a passing gate report on a real cohort; `ssr-replica` reproduces the conditioning signature — conditioned vs. unconditioned distributions diverge as reported, rank stability Spearman > 0.8
1	3–6	`brief`, `agent`, `world` (SurveyRoom), `runner`, `trace`, `analysis`, `report`	`concepts run brief.yaml --fake` completes end to end with no API key; a real run produces a ranked report in < 30 min for < $20
2	7–12	`world` (feed, forum, WOM), `agent` memory + reflection, `population` graph + communities	a herding event in a feed run is explainable by walking the trace from cause to effect; identical seeds reproduce bit-identically
3	13–18	`runner` sweep, `world` remaining recsys modes and intervention kinds, `analysis` anomalies	`sweep run` produces a measured adoption/polarization heatmap with rule-based risk flags

Phase-0 microbenchmark to settle early: tier-A serving for Persona-8B — OpenRouter-hosted vs. self-hosted vLLM vs. a small instruct model with prompt conditioning. Benchmark distribution fidelity, not chat quality; that is the only property that matters here.

11. Performance and acceptance gates¶

Gate	Target	Phase
Coreset decode, 10k personas	< 60 s cold, < 2 s warm	0
Postings filter	never opens a data shard	0
`population.build(n=2000)` under synthetic source	< 5 s	0
SSR rank stability across reference sets	Spearman > 0.8	0
Distribution non-collapse	full 1–5 mass, no point mass	0
`concepts run` end to end	< 30 min, < $20	1
Inference cache hit on baseline re-run	≥ 30%	1
Graph: clustering / giant component / isolates	±0.05 target / ≥ 98% / zero	2
Leiden modularity	≥ 0.4, 4–8 communities of ≥ ~80	2
Determinism	identical seeds → identical `population_hash`, `world_id`, digests	all

12. Deferred, with return criteria¶

Not built now. Each carries the condition that would justify building it, so these are decisions rather than omissions.

Capability	Why deferred	Return criterion
Skill library (VOYAGER-pattern cached reaction templates)	An optimization, not a capability — nothing becomes possible that isn't already. Mining patterns from uncalibrated traces would bake current biases into the engine	≥ 20 completed studies with validated traces, and measured repetition high enough that templated hints save ≥ 30% of tier-A tokens with no distribution-fidelity loss against golden runs
Retail shelf environment	Buy/reject decisions against competitor prices need willingness-to-pay grounding. Shipping synthetic purchase decisions without it produces confident numbers with nothing behind them	WTP grounding available from real data, plus actual demand for pricing studies
Scenario illumination (ASAL MAP-Elites over an archive of runs)	Answers "what outcomes are possible beyond my grid?" — a question worth asking only once baseline runs are trusted and an archive of runs exists to search over. The grid sweep answers the question users actually have first	A benchmark harness exists and passes, plus an archive of validated runs, plus users asking for auto-exploration rather than specifying their own grids
LLM-as-judge scoring	Rule-based anomaly flags cover current needs deterministically and cheaply; a judge adds cost and its own calibration burden	Rule-based flags demonstrably underperform on real traces
Analyst-debate reporting (multi-role bull/bear synthesis)	`analysis` already authors findings deterministically with citations, so the trust invariant holds without it. A debate layer earns its cost only when explanation quality is the bottleneck, not trust	Reports are trusted, and an A/B on decision-usefulness (with vs. without) shows a difference
Corpus-grounded study authoring (ontology drafting + audience proposal, ADR 0014)	The audience-proposal half shipped in M3.5 — `population.interpret_audience` turns words into predicates bounded by the catalog's coverage, through `ChatPort`, with the interpretation shown for confirmation before any draw. What remains is drafting the whole ontology (domains, ordinal scales, relevance order) from the codebook and publishing a versioned diff, which is an authoring surface rather than an engine module	A drafting command that reads the enumerated codebook (attributes, value sets, populated counts — now readable through `HfCoresetSource` and `IndexCoresetCatalog`), proposes a `CategoryOntology`, and writes it as a new versioned file a person approves (ADR 0004)
Packed 4-bit decoder and `HfCoresetSource`	MET by M3.5 (`jahan/ports/decoder.py`, `jahan/ports/hf.py`). The decoder is verified against a generated parquet in the release's real packed format — codes indexed from zero, two to a byte low nibble first, the null bitmap as the sole authority on presence with an absent bitmap meaning fully populated, and `attribute_overrides` beating the code — and against the four cached shards, whose decoded `populated_attribute_count` matches the corpus's own on every row sampled. No corpus row is committed; real-shard tests skip when the cache is absent (ADR 0016)	—
Benchmark harness against real human studies	Requires real human study data, which the engine does not have. Until then every run is `UNCALIBRATED` and the engine claims nothing about accuracy	Human study data available. Building this openly and publishing the methodology is the single highest-value addition to the engine — the `AnchorSource` and `PersonaPatchSource` ports exist specifically so it can be added without changing any module signature

13. Design decisions¶

What changed from the previous 28-module design, and why. Recorded so the reasoning survives.

1. Twenty-eight modules became twelve. The old decomposition followed the pipeline's eight layers, so boundaries fell where data changed shape rather than where complexity could be hidden. That produced modules whose interfaces were about as complex as their implementations, and intermediate types (DecodedRow, MemoryView, undefined "trace views") that existed only to cross a boundary. Modules now correspond to questions — who is in this study?, what does this persona do?, what happened? — not to pipeline stages.

2. Budget enforcement moved into the runner. Previously the governor decided and three modules applied. One policy with three interpretation points cannot be tested in one place, and a misinterpretation degrades fidelity mid-run without failing — a study silently becomes a different study. The governor now applies degradation itself by mutating the run plan; agent and world never learn that budgets exist.

3. Persona conditioning got an owner. Context assembly was referenced by the old spec but assigned to no module, while conditioning is the variable that decides whether the elicitation method works at all. It now lives in agent, where boundary tests assert that the block reaches the prompt and that an unconditioned turn is refused; the effect on answers is measured against a real model, not asserted against a fake.

4. Derivation was split from rendering. Objection clustering previously appeared in both the sweep digest and the report builder, and the trust guard sat in the renderer, testable only by rendering. analysis owns all derivation and the guard; report only formats pre-validated findings and structurally cannot introduce a claim.

5. Ports were introduced at five boundaries. This is what makes deep modules testable without mocking internals — every boundary test runs against in-memory adapters. Two of the ports do additional work: CoresetSource means the engine never depends on the dataset being bundled or licensed a particular way, and AnchorSource/PersonaPatchSource mean calibration can be added later without changing any public signature. That last point is time-sensitive: once these interfaces are published, adding a parameter is a breaking change for every fork.

6. The world modules merged at the interface, not internally. The four OASIS-derived modules present one World port outward while keeping upstream file structure inward. This is the one place where P1 is traded against P6, and the trade is deliberate: upstream OASIS ships fixes worth re-porting, and recsys.py is only valuable while its hot-score math stays diffable against the original.

7. elicitation stays separate despite having one caller. The single deliberate exception to P1, justified by an independent entrypoint, an independent acceptance gate against published data, and its role as the engine's central methodological claim.

8. Nine type-level defects were found and fixed before any code was written. A grilling pass against this document surfaced problems that would all have compiled and produced plausible numbers: one provenance vocabulary serving three unrelated subjects while lacking the word its own contract is named after; "segment" meaning a declared slice, a sampling cell and a discovered cluster, which made sweep files unauthorable; a replicate-seed list that silently produced identical worlds and a structurally zero variance estimate; a dataset whose sparsest rows cannot support the conditioning the method depends on, where the obvious fix biases the population toward synthetic rows; a gate threshold whose pass direction was ambiguous and whose only numeric inputs were the engine's own output; an undefined tick duration alongside a second undefined time unit; per-stimulus reactions that reduced the feed to a slower survey room at triple the cost; a scalar claim-belief that made the engine's most valuable finding unrepresentable; and a per-finding trust ladder still carrying a commercial level with no referent. The resolutions are folded into the sections above; three are recorded as ADRs; the vocabulary fixes live in CONTEXT.md.

9. Phase 8 closed the remaining boundary contracts. A completeness check against this document found module 1 did not yet own every boundary type: Completion, View, WorldDelta, WorldState and the run and sweep results had never been designed, and the ontology lacked a relevance order and anchor-set references. Designing them resolved three latent contradictions — fallback routes that would have failed every trace under ADR 0009's pin check, budget degradation that left no record and so broke both comparability and replay, and guardrail violations with nowhere to go. WorldState and SweepResult were removed rather than designed: world state is rebuilt by replaying the trace, and a sweep is an ordinary run. Recorded in ADRs 0010–0012; tracked as phases 8–12 of the module 1 plan.

Jahan — Engine Architecture¶

1. What the engine does¶

2. Architectural principles¶

3. Module map¶

Mapping from the previous 28-module numbering¶

4. Ports and adapters¶

5. Module specifications¶

5.1 schemas — contracts¶

5.2 brief — intake¶

5.3 population — who is in the study¶

5.4 inference — every model call¶

5.5 elicitation — SSR¶

5.6 agent — the persona turn¶

5.7 world — environments¶

5.8 runner — execution and budget¶

5.9 trace — the audit spine¶

5.10 analysis — deriving meaning¶

5.11 report — rendering¶

5.12 cli — entrypoints¶

5.13 web — the engine over HTTP¶

5.14 ui — the Next.js interface¶