Skip to content

PRD — M3.5 corpus: what is actually in there, and what a field's value is worth

Problem Statement

Three modules are complete, 885 tests pass, and the engine has never read a row of the corpus it was designed around. Reading it changed four things it had assumed.

The example study cannot run. Its conditioning set asks for exercise_frequency, diet_protein_focus and spend_band; those fields are populated on 0.0% of gss rows and 1.3% or less of stackoverflow rows. Nothing in the engine says so until assess() raises "no row in the source carries every attribute the category's conditioning set requires" — a true statement that names neither the field nor the coverage, at the end of authoring rather than the start.

FieldOrigin.GROUNDED claims more than it can support. A gss row's age_bracket is a survey answer; an amazon row's cuis_vegan is a model reading the sentence "Not vegan but still good". Both decode to GROUNDED, so every gate, every audience and every finding inherits a distinction the engine cannot express (ADR 0017).

The sources do not hold what the design assumed they jointly hold. gss measures demographics and no consumption; amazon measures consumption and, on 77.9% of its rows, none of the six demographics that would let the two be joined (ADR 0018).

And the decoder that would have found all of this does not exist. M3's scope deferred it with return criteria — the packed format was documented in a schema this repository had never seen. Those criteria are met: the schema, four shards and the verified decoding rules are in hand.

The risk this module addresses is narrow and demonstrated. Green tests against a synthetic source cannot falsify assumptions the synthetic source was built from. Every defect above survived three modules of review because the only corpus the engine ever read was one that agreed with it.

Solution

One module, one new public function, and a correction pass across the three that exist.

preview(request, *, catalog) -> AudiencePreview answers what the corpus holds, from an index alone — built once from the shards holding the admitted sources, saved beside the cache, and loaded without opening a shard thereafter. Per source: how many rows match, how many carry the attributes at all, which are MEASURED, EXTRACTED or absent, how many fields would be synthesized, which constraints the relaxation ladder would drop, and the resulting evidence grade. It costs nothing and can be run ten times before anything is downloaded (ADR 0020).

HfCoresetSource makes the corpus readable: 4-bit codes packed two to a byte low nibble first and indexed from zero, a null bitmap that is the sole authority on presence and may be absent entirely, and attribute_overrides that supersede the code and are always out of vocabulary.

Around those, four commitments. A field says whether it was measured or extracted. A tier grades a claim and never gates one, with a single refusal: category targets may not be claimed on attributes that are not measured. A shard is never downloaded implicitly. And no corpus row enters this repository.

User Stories

Knowing what is there

  1. As a study author, I want to describe an audience in words and see how many personas match, from which sources, before downloading anything, so I learn what I am getting while I can still change it.
  2. As a study author, I want "nobody was asked this" distinguished from "nobody is like this", so a zero that means look elsewhere is never read as an empirical finding.
  3. As a study author, I want every count reported with its denominators, so a rare audience and an unmeasured attribute cannot look identical.
  4. As a study author, I want to be told which fields would be synthesized and how many, so the share of invented data is a decision rather than a surprise.
  5. As a study author, I want a proposed filter naming a field with no coverage anywhere refused while I am authoring, so an unrunnable ontology fails before a run.
  6. As an analyst, I want the preview to show which constraints the relaxation ladder would drop, so a study that will quietly widen is visible before it does.

Reading the real corpus

  1. As a maintainer, I want the packed decoder to apply overrides over codes, so the 40% of wiki rows whose age_bracket code is a placeholder are not read as toddlers.
  2. As a maintainer, I want an absent null bitmap read as fully populated, so the 400,000 synthetic rows are not silently dropped.
  3. As a maintainer, I want a decoded row's populated count checked against the corpus's own, so a decoding error fails loudly rather than skewing a distribution.
  4. As a user, I want a shard downloaded only when I ask, verified against the manifest's SHA-256 on every use, and cached outside the repository, so a study never pulls a gigabyte on my behalf.
  5. As a maintainer, I want a row's source read from the release's own source column, never guessed from the shape of its identifiers, so an unfamiliar identifier cannot land a row in the wrong — and more trusted — source.

Grading a claim

  1. As a methodologist, I want a survey answer and a model's reading of review text to carry different origins, so what the engine knows is separable from what it inferred.
  2. As an analyst, I want a gate report to carry the weakest tier among its gated attributes, so a pass is never read as stronger than its weakest evidence.
  3. As a methodologist, I want category targets refused on attributes that are not measured, so "matches the real category" cannot rest on an LLM reading product reviews.
  4. As a study author, I want an audience filtering on an extracted or synthetic attribute permitted and labelled rather than refused, so exploration stays possible and honest.
  5. As an analyst, I want a population mixing sources graded by the weakest source in it, so reaching a quota by adding synthetic rows is visible in the verdict.

Studies that run

  1. As a new user, I want the quickstart to run offline with no download, so the engine is usable before any corpus decision.
  2. As a researcher, I want one example study that runs against real rows end to end, so the adapter is proven by use rather than by unit tests.

Implementation Decisions

The catalog seam. A CoresetCatalog protocol separate from CoresetSource: coverage(attributes, sources) and count(predicates, present, by_source=True), answered from an index without decoding rows. Separate rather than three more methods on CoresetSource because the cost profiles differ by four orders of magnitude, and because typing preview() to the catalog is what stops it growing a row read later. SyntheticCoresetSource implements both; in memory the distinction costs nothing.

The decoder. Codes are 4-bit, two per byte, low nibble first, indexed from zero — confirmed 8/8 against entities whose sex is not in doubt, where a one-indexed reading gets 0/8. null_bitmap is the sole authority on presence: populated_attribute_count equals its present-count on 586/586 rows sampled and never the non-zero-code count, and an absent bitmap means fully populated. attribute_overrides supersede the code, sit on code 0 without exception, and are out of vocabulary without exception — 345,483 of 345,483. Most are structured missingness (null, Not applicable, No coding activity) and fold to absent; a minority are real values the enum cannot express (65+, Adult children) and decode as absent, never admitted raw and never guessed into a band, because values() vocabularies are closed and a demographic is never synthesized. They are counted as unexpressible in coverage and the preview, because the loss follows the value — every 65+, never a 25-34 — and a value-correlated exclusion must be visible. The vocabulary is consulted before the missingness sentinels, since 435 codebook values are themselves words like None. The decoder verifies its own output against populated_attribute_count and fails loudly on mismatch.

Downloads. Explicit and opt-in. A study needing an absent shard fails with the command that fetches it; nothing downloads on a user's behalf. Cache lives in a user cache directory, never in the repository, and every file is SHA-256 verified against manifest.json on each use rather than only at fetch. A row's source is the release's own source column. (This section first said the corpus recorded source only in its index and that it had to be derived from identifiers; the column exists, and was missed because only metadata_json had been inspected.)

Tiering. FieldOrigin gains MEASURED and EXTRACTED where GROUNDED stood, derived per field from source and assignment_type by the adapter: a survey source's value is measured where the instrument recorded it (direct, structured_claim, or no grounding entry) and extracted where a model inferred it (summary_inference, unsupported) — 15% of Stack Overflow's att_ai; a text source is always extracted; an unknown source is never measured by default. It is not derivable from either alone: gss and amazon both report direct, and within a source it varies — wiki is 49% unsupported at a median confidence of 1.00, stackoverflow is 92 unsupported in 2.36M entries. The tier grades; it does not gate. GateReport carries the weakest tier among its gated attributes, following the rule GateReport.reference already sets, and the one refusal is CATEGORY_TARGETS on non-measured attributes.

The natural-language step. Through the existing ChatPort, with the coverage table for the admissible sources as the model's entire vocabulary, output validated against the catalog before display, and the interpretation shown back for confirmation as ADR 0014 requires. A filter naming a field with no fill anywhere is refused, not passed through — this is the specific guard against a model proposing spend_band = 10_20 because the schema lists it.

Sources as a parameter. admissible_sources joins PopulationParameters — a study parameter, recorded on the manifest, never read from the environment. A union across sources is permitted and never silent: the preview shows the resulting mix and the weakest-tier rule grades it.

Example studies. The beverage study stays on SyntheticCoresetSource as the offline quickstart, its ontology re-authored so its conditioning set is honest about what a synthetic source supplies. A second study — AI-skeptical senior developers, on stackoverflow — becomes the real-data example, chosen because it is the only source measuring demographics and attitudes on the same person, so its audiences filter real survey rows rather than projections. It grades extracted rather than measured, because part of its attitude fields was inferred — the grade reporting that is the example working, not failing.

Contracts amended. Module 1: FieldOrigin's fourth value and the sweep it implies through Population's invariants and the hash-stability pins; the evidence tier on GateReport; admissible_sources on PopulationParameters; the AudiencePreview contract. Module 2: the ontology's conditioning set is checkable against a source, and the example ontologies change. Module 3: assess() keeps its job unchanged, build() records the tier, and targets_for gains the measured-attribute refusal. SCHEMA_VERSION stays 1.0.0 and unreleased, so the pins are re-pinned rather than migrated.

Testing Decisions

Boundary tests through preview, assess and build. The packed decoder is tested against a generated parquet written in the real format — 4-bit codes, null bitmap, overrides, grounding structs — never against corpus rows, because ADR 0016 makes a subset inherit the corpus's terms and the fixture would redistribute it. The generator is the test's own data, so every decoding case is expressible: an override on every field, no bitmap at all, a code at the vocabulary boundary, a populated_attribute_count that disagrees.

Integration tests against the real shards run only when they are present locally and skip otherwise, so the suite stays offline and the socket refusal in tests/conftest.py is untouched. Those tests are what prove the four shards decode as measured rather than as assumed.

The tests carrying the most weight are the ones this module exists for: an override-bearing row decodes to its override and not to code 0; a row with no bitmap decodes all 1,290 fields; a preview reports 0 / 0 / n and 0 / m / n distinguishably; a filter on a zero-coverage field is refused at authoring; a CATEGORY_TARGETS claim on an extracted attribute is refused; a population mixing sources grades to the weakest; and no shard is read by any preview() call.

Out of Scope

The real model transport, which is module 5 and which this module reaches only through ChatPort and its fake. The Stack Overflow holdout that would measure how much attitude variance demographics actually explain — it needs a real projection model, and ADR 0019 records it as owed rather than done. A second corpus with measured demographic-and-consumption joints, which ADR 0018 leaves as the route to a genuine FMCG claim and which is a separate adapter behind the same port. Donor fusion, donor provenance and a DONATED origin, designed before the bridge-variable measurement and rejected by ADR 0018. Persisting a population, unchanged from M3's deferral. Sampling proportional to category targets, likewise.

Further Notes

SALVAGE.md needs its MatrAIx section reconciled against the shards: the source counts, the per-shard segregation, the index files that become the catalog's backing, and the decoding rules the dataset card omits. FINAL_ARCH.md §5.2–5.3 needs the tier, the catalog port and the two-stage pre-flight; §12's return criteria for the decoder are met and the section should say so. CONTEXT.md gains Measured, Extracted, Evidence Tier, Audience Preview and Coverage. ADR 0002 gains the coverage consequence and ADR 0014 the catalog-constrained vocabulary; neither is superseded.