Fixed
- The explorer's entire application shell crashed on the home page
—
TypeError: Cannot read properties of undefined (reading
'toUpperCase'), surfaced as "Stellar Index hit an unexpected error"
on every route.
A Soroban contract asset has no code; the handler field carries
json:"code,omitempty", so the key is omitted entirely for those
rows. But the OpenAPI spec marked code required, so every
generated type declared code: string — non-optional — and
coin.code.toUpperCase() type-checked cleanly while receiving
undefined at runtime. The spec and the generated types agreed with
each other and both disagreed with the server: the same class as the
display_decimals gap (F-SDK-04).
It stayed latent because /v1/assets silently ignored order_by, so
the home page got the top ten by ALL-TIME observation count —
classic-only in practice, every row carrying a code. Once the ranking
was corrected to the trailing-24h volume its caption always claimed,
four Soroban contracts entered the top ten and the page threw inside
the root layout, escaping to global-error.
code is now optional in the spec, which turns this class into a
compile error rather than a runtime crash — it immediately surfaced
nine unguarded dereferences, including on the asset detail page
and in the search modal, all fixed. Code-less assets now render a
truncated contract id instead of a blank cell. A regression test
renders the exact production row shape and is proven red against the
pre-fix component with the identical error.
Added
- An apply path for the r1 ansible config.
deploy-binary.yml
swaps binaries only, so the config-apply gate in deploy.yml was a
forcing function with nowhere to force *to*: the full render needs
the ansible vault, which lives in Actions secrets and nowhere a
person can reach from a laptop. Config therefore lagged binaries and
features shipped dead — exactly what that gate exists to prevent
(the 2026-08-25 declared-peg and rules.d incidents). The same
credentials that already prove drift every Monday can now correct
it: ansible-drift.yml takes an apply input that drops --check.
Deliberately opt-in per run. The scheduled run and every default
dispatch stay --check, so the workflow remains a detector; only an
explicit apply=true mutates r1, and the drift verdict is skipped
only on that arm (an apply is *expected* to report changed>0). The
input's description states plainly that applying restarts the
indexer, aggregator and api.
Fixed
- The actions SHA-pinning lint matched `uses:` as a substring, so
ordinary prose in a workflow tripped it: an error message reading
"Usual causes: sshd not listening…" was parsed as a tag-pinned
third-party action named
sshd and hard-failed the PR that
introduced it. The match is now anchored to a YAML key boundary
(line start, whitespace, or list dash). Verified in both directions —
a genuinely tag-pinned action is still caught, and prose containing
"causes:" no longer fails. The capture index moved with the added
group, which would otherwise have made the lint report the wrong
token.
- A failed test-net deploy named neither hop. Reaching a NAT-only
test-net VM goes through a ProxyJump, and BOTH hops must have their
host key pinned in the one
known_hosts file the workflow writes.
When the jump host's key is missing — or the deploy key isn't
authorised on it — ansible fails with UNREACHABLE! ... "Connection
closed by UNKNOWN port 65535", which identifies neither hop and
gives an operator nothing to act on. Both test nets failed exactly
that way on 2026-08-31 (#434) while r1, which has no jump, deployed
cleanly. deploy.yml now preflights the SSH path before ansible
runs: it checks that the jump host and the target are each pinned,
probes the jump hop on its own, and reports which one is broken.
Host names and counts only — never key material, and any
key-shaped token in the captured stderr is redacted.
Fixed
- `release.yml` failed the whole release when the CHANGELOG section
exceeded GitHub's release-body limit. GitHub caps a release body at
125,000 characters and returns HTTP 422 "body is too long" — *after*
every binary has been built, so the run fails at the final step with
nothing published. v0.51.0 hit exactly that: its section extracted to
137,450 characters, because v0.49.0 and v0.50.0 shipped without
CHANGELOG sections of their own and the next promotion absorbed
everything back to v0.48.0. The notes are now truncated at a line
boundary under the cap, with a link to the full section in
CHANGELOG.md — the notes are a convenience copy, the CHANGELOG is
the record, and a release that publishes with a pointer beats one
that does not publish at all.
Fixed
- Five guards that did not check what they claimed — all mine, all
found by an adversarial review of the 2026-08-31 merge range, each
fix proven by mutation.
- `lint-rule-structure.py`'s metric-label regex crossed declaration
boundaries. A non-greedy gap scanned past a label-less
NewGauge/NewCounter until it found the NEXT metric's
[]string{…} and credited it to the wrong metric — 19 metrics
affected. So the new fixture-realism check ran against fabricated
"declared" sets and its error message named labels the emitter does
not have: stellarindex_anomaly_freeze_active is a bare NewGauge
with zero labels, yet the lint credited it with {op} and passed a
fixture writing a series production can never emit.
- `lint-unit-failed-baseline-test.sh` was executed by nothing. I
wrote it, its commit trailer claimed verify.sh ran it, and it was
wired into neither verify.sh nor CI — and check-verify-parity.sh
only enforces CI→verify, so a script in *neither* is invisible. Now
wired to both. Its check was also one-directional: it walked
baseline→regex only, so a unit added to the exclusion regex with no
baseline entry was silently exempted from the catch-all and had
no dedicated alert. The reverse direction now fires.
- `lint-deploy-systemd-authority.sh` asked two independent
questions. "Is - <unit> a list item anywhere in tasks/" AND "does
deploy/systemd/{{ item }} appear anywhere in tasks/" — the second
being a repo-wide constant satisfied by one unrelated task. So
classification collapsed to the first, and adding a unit to an
ansible.builtin.systemd enable loop (installing nothing) passed
as authoritative — the exact "enable a unit you don't install"
footgun the lint's own header describes. Both facts are now required
of the SAME task, via a parser rather than greps.
- The orphan-branches footer never rendered when it was the only
thing to report. The header promised dispositioned branches are
"still COUNTED, in a footer line", but the issue step was gated on
the orphan count alone — so in the header's own worked example (the
17 landed fix/issue-* branches clearing the grace with no other
orphan) the close step fired instead and the count appeared only in
a job log.
- The CS-017 freshness test re-implemented the rule instead of
calling it. Deleting the > r.freshnessWindow() term left the
suite green while /v1/price resumed serving months-old buckets
with stale=false. The rule is extracted to
storePriceReader.bucketIsStale and the test now calls it;
deleting the term fails the suite.
- Every R1 deploy was failing its config-apply gate and skipping the
post-deploy smoke test.
deploy.yml read the host's version
sidecars with cat …/deployed-versions/stellarindex-*, but those
files are written by ansible.builtin.copy: content: "{{ version }}",
which writes no trailing newline. Six binaries therefore
concatenated into one token (v0.46.1v0.44.7v0.28.1…); the
^-anchored version filter matched it and sort -V | head -1 passed
it straight through. The gate could not resolve that to a commit and
failed CLOSED — so the binaries deployed fine, the job went red, and
the Served-path smoke step (which has no if:) never ran.
Now reads with awk 1, which emits each record with a trailing
newline whether or not the file had one, and the version filter is
anchored at BOTH ends so any future malformed sidecar is rejected
rather than mistaken for a version.
The gate's self-test gained three checks, each proven to fail on its
own defect: the newline-safe read, the end-anchored filter, and — a
separate latent hole — that the baseline is read before the
playbook. That last one previously asserted the ordering in its
message while only grepping for two strings independently, so moving
the baseline step below the deploy would have kept it green while
making the gate permanently vacuous (the "live" version would be the
version just deployed). - A withholding guard I deleted let the MSP-07 regression back in.
The review sweep replaced
TestPriceServingSeamsAreGated's weak
two-entry subject list with a derived one — correctly — but removed
TestWithholdingGatesAreSpelledOnlyAtTheChokepoint in the same
change, while that commit's own message said *"the MSP-07 half (drift
WITHIN a seam) was always real"*. main.go went on citing the
deleted test by name.
The two guards answer different questions. The surviving one asks
"does every serving seam consult the gates at all" — a method with two
arms satisfies it as soon as ONE arm calls priceWithheld(). The
deleted one asks "is the withholding decision spelled in exactly one
place", which is the only way to catch a single arm drifting.
Verified: reverting the last-trade arm to !r.substance.Allowed(…) —
the literal MSP-07 code, which drops the scam gate — passed CI
before this restore and fails by file and line after it. That
regression matters because an operator setting
disable_substance_gate=true to diagnose a coverage complaint would
silently publish a directory-flagged issuer's last trade as its price,
reversing an owner-level trust decision they never touched.
- The SDK's spec-coverage table was bound to nothing (wave-D
F-SDK-10).
TestSDKCoversSpec reconciles coveredOperations against
the OpenAPI spec, so it catches an endpoint the SDK forgot — but it
cannot catch the table drifting from the SDK. Renaming or deleting a
Client method left the table naming the old identifier while the
reconciliation stayed green. A new guard binds it both ways: every
tabled sdkMethod must resolve on *Client, and every exported
*Client method must appear in the table or in an exemption set with
a stated reason. The second direction is the load-bearing one —
without it the table only ever describes the subset someone
remembered to add.
docs/audit/recipe.md also recorded pkg/client as never
adversarially audited with no behavioural coverage. It was audited on
2026-08-04 and has 156 tests at 78.6% statement coverage, and
"retries" cannot be a gap because there IS no retry layer. Corrected,
and pointed at where the SDK is actually weak. That entry was itself
an instance of the class it now warns about.
- Three price reads served ONE of the two stored market directions
(wave-D UNAUTH-DOS-9). The decoder files each trade in the venue's
observed base/quote ordering and deliberately does not normalise, so
a two-sided market lands in the CAGGs as BOTH
(A,B) and (B,A)
rows — as dirVWAP's own doc says, "every serving read has to fold
the two together itself". Three did not:
- RecentClosedVWAP1mForPair (`/v1/oracle/prices`) dropped every
minute the market traded only the other way. The SEP-40 series went
silently sparse, and for a predominantly-flipped pair the endpoint
returned 200 [] for an asset /v1/oracle/lastprice priced
without difficulty — two endpoints on the same declared SEP-40
surface disagreeing about whether the asset had any history.
- ClosedVWAP1mAtOrBefore (`/v1/assets/{id}` `change_24h_pct`)
understated the 24-hours-ago anchor two ways: for a two-sided
bucket it returned one leg's VWAP as if it were the bucket's,
ignoring the other leg's volume; for a flipped-only bucket it
matched nothing and the percentage vanished.
- TimedVWAPs1mForChangeSummary (the change-summary worker), the
same defect in the aggregator's series read.
All three now read both orientations and fold them with the exact
volume-weighted union combineDirVWAP already serves — the
inversion stays in Go, never in SQL, where 1.0/vwap would round
the flipped leg before it was weighted (ADR-0003).
This is the same bug for the third time: LatestClosedVWAP1mForPair
was fixed for it in audit-2026-07-23 (MNY-06), and that fixer
reported RecentClosedVWAP1mForPair as R-076 — recorded in the
audit's remediation state and never dispositioned. Each earlier fix
shipped a test pinning that one function's query, and none could see
the next reader. The new guard is written against the CLASS instead:
it parses every string literal in the package and fails on any
pair-bound CAGG read that filters a single orientation. It found the
third instance immediately — one that neither the finding nor its
skeptic had spotted — and it fails loudly if its own subject set ever
comes back empty. - `/v1/vwap` and `/v1/twap` served a scam-flagged issuer's aggregated
price, and the guard's own docs said they didn't (wave-D
MSP-02/EXR-04). Every other price surface withheld it —
/v1/price, /v1/price/tip, /v1/price/batch, the SEP-40 oracle,
the asset headline — while these two returned 200 with a number.
Reproduced live against a directory-flagged issuer before the fix.
The gap was *documented as fixed*: pricingguard/scam.go claimed the
gate sat "at the price-reader seam so every reader-backed surface
(…, /v1/twap, /v1/vwap, …) is covered by ONE gate", and PR #182's
merged body repeated it verbatim. Neither endpoint goes through the
price reader at all — both compute from raw trades via their own
fetch — so the claim was never true, and no test contradicted it. The
doc now states the six real call sites and says plainly that no
single seam covers them all.
Gated in the two HANDLERS, deliberately not in the shared
tradesInRangeWithStablecoinFallback: that helper is also the fetch
behind the single-bar /v1/ohlc, which the guard's own docs, the
config reference, and the withheld problem's own guidance text all
promise stays visible — gating there would make our error message's
escape-hatch advice a lie.
Scam gate only, not the substance gate. The scam gate is targeted
(flagged issuers) and directly implements the 2026-08-25 decision.
Applying the substance gate here would newly 404 every *thin* pair —
a breaking change, and arguably wrong on principle, since ADR-0015
and VWAPResult's own doc position /v1/vwap as the "narrow the
window and compute it yourself" surface *opposite* /v1/price. That
is an owner decision, not something to smuggle in with a scam fix.
- A withheld price verdict reached on the stablecoin-proxy leg was
swallowed, so the API said "no price data" (wave-D MSP-06). The
direct fiat read misses for the dominant on-chain shape (trades are
quoted in issuer stablecoins, not
fiat:USD), the handler enters
priceFallback, the proxy loop's LatestPrice(asset, <peg>) returns
ErrPriceWithheld — and the loop's bare continue, written to skip
an INACTIVE peg, discarded that verdict along with the miss. The
response was errors/price-not-found.
Those are different answers. "No price data" tells a customer to look
nowhere; the withheld problem names /v1/observations, /v1/ohlc and
/v1/history, where the data IS available — that guidance is the
whole reason the distinct problem type exists. Same swallow affected
/v1/oracle/lastprice and /v1/oracle/x_last_price.
Withheld is now sticky, not terminal: a later peg that can serve
still wins (a real price beats a 404), and the verdict surfaces only
when nothing serves. The batch path keeps dropping it, deliberately —
that envelope has no per-row problem shape.
The two existing withheld tests were left untouched. They are
load-bearing — mutation testing confirms deleting the direct-read
short-circuit fails both — and the finding's suggestion to rewrite
them would have deleted live coverage.
- `stellarindex_oracle_stale` could never fire, for any oracle —
and its test passed anyway (wave-D ALERT-02). The rule compared
stellarindex_oracle_last_update_unix (labels {source, asset})
against stellarindex_oracle_resolution_seconds (labels {source})
with no on()/ignoring(). A vector-to-vector operation requires
*identical* label sets, so no pair ever matched and a silent oracle
raised nothing. Now joined with on (source) group_left(), which
keeps the left side's per-asset cardinality so one silent asset still
tickets.
The promtool case covering it was green only because it fabricated an
asset label on the resolution series — a label the emitter is
structurally incapable of producing (WithLabelValues with two
arguments on a one-label vector panics). With a realistic series the
rule returns nothing at all, which is what the corrected test now
proves. A second case pins the fan-out, so a regression to a bare
comparison fails rather than silently matching nothing. - Both usd_volume coverage alerts went silent at exactly 100% NULL —
their worst case (wave-D ALERT-04). Each divides a "priced" counter
by a "total" counter, but the
usd_volume_populated="yes" child is
created only by the first *priced* insert (there is no zero
pre-initialisation). So when a source prices nothing at all, that
series does not exist, an aggregation over it is the empty vector
rather than zero, and the division yields no sample: total pricing
failure was the one condition these alerts could not see. Partial
failure fired correctly, which is why it was never noticed.
Both arms now substitute an explicit zero for a source present in the
denominator, so 0/N = 0 and the alert fires. Neither alert had *any*
promtool coverage; there is now a test file for both, including
full-coverage controls so the fix cannot over-fire. Proven red against
the pre-fix rules — both cases returned no alert whatsoever. - A scam-flagged issuer still published an all-time-high dollar price,
and a declared-peg asset still served the price series the listing
refused (wave-D MSP-05 / MSP-04).
suppressScamIssuerPricing nulled
six fields but not ath, so a directory-flagged token returned
"price_usd": null next to "ath": {"usd": "0.0091"} — a published
USD valuation, from the same USD-quoted CAGG, for an asset the
platform had just decided must publish none. Separately, the listing's
sparkline rule and the detail path's series suppression asserted the
same product question in two places and drifted: the listing excluded
declared-peg rows, the detail path did not, so /v1/assets drew no
sparkline while /v1/assets/{id} served a full
price_history_24h/_7d charted from the dust market the substance
gate had refused.
Both paths now share one predicate, priceSeriesPublishable, which
answers "may this payload carry a derived price-over-time claim?" for
sparklines, price_history_* and ath alike. The peg price itself is
untouched — the peg is the published claim; only the market series
charted beside it goes. - Two more alerts that could not fire (wave-D ALERT-03, ALERT-06),
both instances of the same class as #389/#390 — an expression nothing
evaluated against what the emitter can actually produce.
stellarindex_external_poller_error_rate_high summed
rate(…{outcome="success"}) + rate(…{outcome="error"}) with no
matcher. PromQL's one-to-one matching compares the full label
signature and outcome differs on every candidate pair, so the +
yielded the empty vector unconditionally: the alert could not fire at
any error rate. Now sum without (outcome) joined with
ignoring(outcome), which keeps job/instance for the annotation.
The success|error selector is deliberate — a third outcome,
skipped (post-429 cooldown), is excluded on purpose, and summing all
outcomes instead would leave the alert dead during throttling, the
exact incident shape it exists for.
stellarindex_divergence_no_reference and
…_refresh_error_dominant compare a failure outcome's rate against
the ok outcome's, but stellarindex_divergence_refresh_total was
the one alert-referenced outcome counter missing from
seedBoundedLabelSeries. A counter child does not exist until its
first increment, so an aggregator that had never completed a
successful refresh — every reference unreachable and the process
restarted mid-outage, as deploys routinely cause — had no ok series,
making both comparisons empty and both alerts silent while
flags.divergence_warning served frozen and a live depeg went
unflagged.
The seed guard derives its subject set from the emitter source, so a
new outcome added without a matching seed fails on the day it is
written. - **The verify-archive staleness page measured when the timer last
*fired*, not when verification last *succeeded*** (wave-D ALERT-10).
stellarindex_verify_archive_run_stale is a severity: page
guarding R1's role as integrity leader (ADR-0016) — the nightly
chain verification R2/R3 trust. It read
node_systemd_timer_last_trigger_seconds, which systemd updates on
every firing regardless of how the triggered service exits. So a job
that failed every single night kept the gauge perfectly fresh, and
the page was defeated by exactly the scenario its own description
names: *"either the timer isn't enabled or every recent attempt
failed."* The Tier B ticket had the same defect.
Both now read stellarindex_verify_archive_last_success_unix, a new
per-tier gauge the verify-archive binary writes into its existing
node_exporter textfile and advances only on a clean exit; a
failed run carries the prior value forward. A host that has never
completed a run emits an explicit 0 rather than no series at all.
The runbook had already documented the limitation as a known caveat
— a NOTE under Symptoms and a state-table row marked "Shouldn't
happen". That row describes a reachable, expected state now, and both
are rewritten. Writing a limitation into a runbook makes it
survivable; it does not make the page work. - The total-ingestion-loss SEV-1 sent a RESOLVED while ingestion was
still completely down (wave-D ALERT-01).
stellarindex_ingestion_all_sources_stopped matched on
sum(rate(stellarindex_source_events_total[5m])) == 0. When the
indexer *process dies* the series stops: for a few minutes its last
samples are still inside the range window and rate() is 0, so the
page fires correctly — then the samples age out, the series vanishes,
and sum(rate()) over nothing is the empty vector rather than zero.
The alert stopped matching and Alertmanager resolved a P1 that was
still fully in progress, which reads to a responder as "it fixed
itself". It also never fired at all when the indexer was already down
as Prometheus started, or when Prometheus restarted mid-outage.
An absent_over_time branch now takes over at exactly the point
rate() gives up — both use a 5m window — so the page persists until
events resume. This adds no new paging scenario: the rate branch
already fired after 3m of silence, so any restart longer than
for: 3m pages today. What changes is that the page no longer lies
about recovery.
The existing test could not have caught it — its fixture is a flat
counter, which is a stalled source with a *live* process, not a dead
one. A truncated-series case now covers the process actually dying.
- A percent-encoded slash forged the SSE exemption, so 13 routes could
be asked to run with no request deadline at all (wave-D
UNAUTH-DOS-4).
RequestTimeout exempts streaming endpoints by the
/stream path suffix, but tested that suffix against r.URL.Path —
the *decoded* path — while Go's mux routes on the escaped form. Those
disagree exactly when a wildcard segment contains %2F:
GET /v1/assets/native%2Fstream routes to the ordinary
/v1/assets/{asset_id} handler while its decoded path ends /stream.
The exemption now keys on r.URL.EscapedPath(), which is what the mux
itself routes on, so the two cannot disagree about what a request is.
Guarded by a sweep that enumerates the routes from server.go rather
than listing them: the forgery worked against every trailing-wildcard
route, and new ones are added regularly, so a hand-written table would
pin today's routes and miss tomorrow's. It found 13 pre-fix, and
checks the converse too — genuine SSE routes must *keep* their
exemption, since a fix that bounded the streams would be worse than
the bug. - The SEP-40 `prices()` closed-bucket read was not sargable (wave-D
UNAUTH-DOS-3): it applied a function to the indexed column
(
bucket + INTERVAL '1 minute' <= now()), so the planner could
neither use the bucket index nor prune chunks at plan time. Rewritten
to bucket <= now() - INTERVAL '1 minute' — semantically identical,
no change to what is served. Its sibling combined-direction template
already had the correct form.
It drifted because it was an inline const q inside the function
body, invisible to the package's existing sargability guards, which
assert over package-level templates. Hoisting it is the durable half
of the fix and is what makes a guard possible at all.
Deliberately *not* given a literal lower bound or the 14-day existence
gate its neighbours use: both change what a documented public endpoint
serves (a dormant asset's last N closed buckets becoming an empty
array), which is an owner decision rather than a query-shape fix.
- A TimescaleDB counter built to make silent CAGG starvation
alertable had no alert (wave-D ALERT-12), and the lint that should
have noticed could not see its emitter (ALERT-07's class).
stellarindex_timescale_job_failures_total exists because r1 once
failed 37–69% of every CAGG refresh run with failed to start job —
background-worker starvation — and nothing surfaced it: the jobs got a
slot on a later tick, so last_run_status read Success, the caggs
were never stale, and both existing alerts stayed correctly quiet. The
only evidence was this counter, which no rule referenced. Now
stellarindex_timescale_job_failures_climbing (informational, >10
failures in 6h sustained 30m) with a runbook that branches on the
actual err_message, since failed to start job means starvation
rather than a broken job body.
Adding it exposed the lint problem. lint-metric-refs rejected the
rule as referencing a metric "nothing emits" — but marking it
KNOWN_INERT would have been false: it is emitted and scraped every
60s. is_emitted() grepped only *.go/*.sh/*.prom, and several
textfile probes live as inline ansible content: blocks, so their
real metrics had all been parked in KNOWN_INERT with comments
reading "NOT inert: the probe timer runs every minute on r1". That
made the list mean two incompatible things — "no producer exists" and
"the producer is invisible to this lint" — which is the confusion the
gate exists to prevent.
The lint now scans ansible tasks//handlers/ YAML, scoped so a rule
file cannot satisfy its own reference, with comment-stripping intact
so a metric named only in a comment still counts as dead. Its own
stale-inert check then flagged 7 metrics wrongly listed as inert,
including the galexie-catchup and stellar-stack-version probes; all
seven are removed. KNOWN_INERT means "no producer" again. - A promtool fixture could assert against a series production cannot
emit, go green, and certify an alert that could not fire. That is
not hypothetical:
stellarindex_oracle_stale was unfireable for every
oracle while its test passed, because the fixture wrote
stellarindex_oracle_resolution_seconds{…,asset="XLM"} and that
metric is declared with one label — WithLabelValues with two
arguments panics. lint-rule-structure now checks every rule-test
series: label set against what the emitter can actually produce.
The declared set is the union over *every* emitter shape, not just the
Go vector: verify-archive declares {chunk_idx, reason} in-process
but production scrapes its node_exporter textfile, which carries
{tier, reason} — checking against the Go declaration alone flags
correct tests. Labels attached by the scrape (job, instance, and
r1's static binary) are read from the scrape config rather than
hardcoded, so a new target label does not turn the lint into a source
of false failures.
It found two real violations: a second copy of the asset="XLM"
fabrication, in a negative case that passed either way — which is
exactly why it survived when the firing one was corrected — and three
stellarindex_trade_inserts_total{usd_populated="true"} fixtures
inventing both a wrong label name and wrong values
(usd_volume_populated, yes/no).
A rule-test *coverage-percentage* gate was considered and rejected: it
would have caught none of this wave's unfireable alerts, and it
creates pressure to shrink a baseline by writing more fixtures — the
mechanism that produced the false greens in the first place. - Three ways a malformed `/v1/assets` cursor got past validation
(wave-D KP-2), one of which failed silently.
isNumericPrefix
required no digit, so a volume prefix of . validated and was bound
as $n::numeric for Postgres to reject; the rank tier was checked
only for being digits, so 2147483648 reached an int4 placeholder;
and an over-int64 observation-count prefix passed, then made
parseAssetCursor degenerate the whole cursor to (0, 0, "") — which
matches no rows, so the client got a 200 with an empty page,
indistinguishable from end-of-pagination. All three are now rejected
at the boundary with a 400. markets.go carried a hand-copied
duplicate of the same digit-less loop and now shares the helper.
- A page whose rows were all folded away stopped pagination dead
(wave-D KP-3). The next-cursor was emitted only when
hasMore && len(out) > 0, but out shrinks *after* the query —
suppressCatalogueTwins drops rows and foldAliasTwins collapses
them. A page that folded to empty therefore emitted no cursor while
more rows remained, and the walk ended early. The cursor is built from
the raw last row, which exists whenever hasMore is true, so it is
now emitted on hasMore alone.
- The config-apply gate diffed against the wrong baseline, so a
catch-up deploy was told "the binary deploy is complete" (wave-D
LID-5).
deploy.yml called config-apply-gate.sh with two arguments,
omitting the host's live version — so the gate fell back to "the
previous release tag by ancestry", which is only correct when the
fleet is exactly one release behind. deployed-versions.md states
plainly that a tag cut does not imply the fleet moved to it, and 8 of
23 adjacent tag hops over v0.40.0..v0.49.0 have an empty
config-surface diff — so the green branch is reachable.
Measured on real tags: the two-argument form reports *"no
config-surface changes between v0.47.1 and v0.47.2 — the binary deploy
is complete"* and exits 0, while the three-argument form against a
real host baseline of v0.45.0 finds 13 changed config surfaces and
exits 1.
The workflow now reads the host's deployed-versions sidecar *before*
the playbook runs — afterwards it reports the version being deployed,
which would make the baseline trivially equal and the gate vacuous —
and takes the lowest version across the managed binaries, since config
is unapplied if any binary predates it. Best-effort: an unreachable
sidecar falls back to the documented ancestry default with its
existing warning, because failing the deploy there would trade a
weaker gate for an outage risk.
The script and its self-test were always correct; only the caller was
wrong. So the new guards check the *caller* — a script-level test
cannot catch a caller-level omission. - The binary version-skew probe scored an absent binary as perfectly
healthy, and its alert claimed to cover that (wave-D LID-2). The
probe globs the install directory, so a file that is not there is
never visited: an entirely absent release binary yielded
skew=0
*and* probe_success=1. A present-but-non-executable one was skipped
just as silently by [ -x "$path" ] || continue — while the alert
description named "missing, not executable" among the causes it
covers. Two of its four named causes were false.
The non-executable half is fixed: a present, managed, non-executable
binary now marks the run degraded. The parked-build skip
(.prev-/.rolledback-) moved above the executable test, so a
non-executable parked artifact cannot be mistaken for a broken live
binary.
Absence is deliberately *not* fixed here. The obvious approach —
asserting all six managed binaries are present — would pin the alert
permanently red on testnet and futurenet, where fewer are installed by
design, and the probe's own comment already warns that "a
permanently-firing alert is the same as no alert". Detecting absence
needs a host-derived expected set, not a hardcoded count. The
description now states plainly what the probe cannot see and why the
naive fix is wrong: an alert that overstates its coverage is worse
than one with a documented gap, because the first makes you stop
looking.
- An oracle row whose stored asset/quote text would not parse vanished
from the served stream with no signal (wave-D SI-OC-04).
LatestOracleStreams dropped it with a bare continue — no log, no
metric, no error — so it was simply absent from /v1/oracle/streams
and the explorer's /oracles page.
The silence mattered most exactly where it was most likely: the
documented remediation for a mislabelled oracle row is an
operator-run raw SQL UPDATE against that column, which has no
CHECK constraint. A typo therefore deleted the row from the served
surface rather than erroring — and the operator would watch it
disappear and reasonably conclude the relabel had worked. Now counted
by stellarindex_oracle_stream_rows_unparsed_total{source,field},
with a ticket alert and a runbook whose diagnosis leads with the
operator-typo shapes (missing prefix, truncated strkey, rwa:/raw:
confusion).
Deliberately *not* pre-seeded, unlike the divergence outcome counters:
seeding matters when a rule compares two children of one metric or
divides by one, and this is a bare threshold on a single series that
appears exactly when the bad thing happens.
- The explorer showed an assumed oracle quote as though it were
observed (wave-D SI-OC-01). An unmapped symbol carries no reliable
denomination, so the decoders record a default —
fiat:USD unless the
symbol ends in a recognised fiat suffix. The unmapped panel rendered
that default as a linked quote beside the price, so a hypothetical
wstETH/ETH or bare wBTC_FUNDAMENTAL would display "USD" next to a
number that is not dollars. The column is now labelled *assumed*, is
no longer a link, and carries a hover saying it is a decoder default;
the OpenAPI description says the same for API consumers.
Only the presentational half is changed. Broadening the suffix rule
substitutes one guess for another and contradicts the design doc, and
the bare-suffix case is not inferable at all — mapped: false already
means the denomination is unknown by design, and that is the contract. - `deploy/systemd/` is mixed authority, and four units in it are
installed by nothing (wave-D LID-7). Ansible copies
config-assertions.{service,timer} straight out of that directory, so
editing those changes production — while every other file is either a
reference copy shadowed by a .j2 in the role (editing it changes
nothing, which is how several drifted) or an orphan that nothing
installs at all. r1-deployment-state.md documents an operator
convention of scp-ing units straight from here, which makes an
undeclared orphan a live footgun: it looks like a deployed unit, it
starts if copied, and nothing ever reconciles it.
A new lint classifies all 23 unit files as installed, templated, or a
declared orphan, and fails on anything else — so the three states stay
distinguishable instead of being folded into "files in a directory".
The four orphans are declared with reasons rather than deleted:
ch-live-catchup.{service,timer} are the ClickHouse lake's only
self-healer and their absence is a real gap (LID-1), while
stellarindex-completeness.{service,timer} are superseded — their
ExecStart target does not exist anywhere in the repo and the role
ships compute-completeness for the same job.
- `ch-rebuild -write` leaves the completeness verdict carrying a stale
clean claim over the range it just rewrote (wave-D CV-1).
projector-replay records a projection dirty window so the next
compute-completeness re-reconciles the rewound range; ch-rebuild
records nothing, so the nightly verdict keeps its prior clean claim.
It now says so, loudly, at the point of use — an operator running
-write is told to note the window and re-check the affected sources'
reconcile before trusting the next /v1/coverage verdict.
The automatic record is deliberately not implemented yet, and the
reason is measured rather than cautious: one source's dirty window
(aquarius) already blew the reconcile pass's 120-minute deadline and
needed a bespoke prefilter, against a 180-minute service timeout.
Recording windows for the eight sources the rebuild script drives over
~12.9M ledgers would force the next nightly to re-reconcile all of
them un-prefiltered — a likely timeout that takes out every
source's verdict, which is worse than the stale claim it fixes. That
needs a bounded per-window re-reconcile and a re-measured pass
wall-clock, neither of which can be established without the real lake. - A completeness dirty window could be cleared on evidence that
predates the rewrite it certifies (wave-D CV-6). The clear's bounds
(
from_ledger >= $2 AND to_ledger <= $3) correctly protect a
*widened* window — a concurrent replay that grew the range leaves a row
outside the bounds, which survives — but not a subset re-record: a
replay re-recording the same or a narrower range leaves both bounds
satisfied, so the delete erased evidence of a rewind the run never
verified and the next verdict carried a clean claim over it.
The guard was already provisioned: migration 0125 declares
updated_at and the upsert maintains it. Comparing it makes the clear
optimistic concurrency — it succeeds only if the row is the one whose
obligation this run discharged. Any re-record, wider or narrower or
identical, bumps the timestamp, the delete matches nothing, and the
window stays pending. Fail-closed, costing at most one extra
reconcile.
The bounds are kept: they and the timestamp cover different races, and
a test asserts both survive. Two alternatives were rejected on the
finding's own analysis — tightening the DELETE to equality changes
nothing (a subset re-record leaves both bounds identical), and gating
on the live cursor would make windows recorded at or above it
permanently unclearable, which is the stale-window treadmill this repo
already suffered.
- Two oracle-path comments asserted things the repo's own evidence
contradicts (wave-D SI-OC-05). The external dust-floor constant's
docstring claimed
$1 is an upper bound across the fiat allow-list.
That was true when written (32 codes, GBP/CHF ≈ $1.3 the richest), but
the list was later widened to 133 and brought in KWD ≈ $3.26,
BHD ≈ $2.65 and OMR ≈ $2.60.
The direction of the resulting error was also stated backwards, and is
corrected: under-stating the reference *over*-states the floor, so a
$3.26 KWD leg gets a floor ~3.3× stricter than intended — the
size-biased-dropping direction the constant exists to prevent, not the
harmless one. The exposure is bounded at compile time (every fiat
quote leg a streamer can see is hard-coded, all ≤ ~$1.35), so the
single constant stays; a real FX table would be false precision for an
order-of-magnitude threshold and would rot.
A reflector test comment also stated magnitudes the repo's own
captures contradict (VES 7.3e-6 / XAU 4,100 against a real 2.07e-3 /
4720.90). The constants are left alone deliberately — the decode path
has no magnitude-dependent branch, so restating them buys no coverage
— and the comment now says so and points at the real-fixture test.
The behavioural half of the proposed remedy is rejected: returning
no-floor for an unvetted fiat resolves to 1e-8 whole units, which
*disables* the dust guard for that leg — strictly worse than a
too-strict floor. A third claim, that a ParsePair comment
misdescribes the accepted grammar, is refuted: the comment cites
api-design.md §3 specifically and is accurate about that document.
- Canonical `rwa:` assets linked to pages that do not exist (wave-D
SI-OC-02). ADR-0028 ids like
rwa:XAU fell through to the bare-code
branch, producing /assets/rwa%3AXAU — the API 404s on the prefixed
id and 400s on the bare code, so *both* spellings are dead. They now
render as a plain label with the full id in the tooltip, unlinked.
Deliberately not "strip the prefix like fiat:/crypto:": that would
produce /assets/XAU, /assets/BENJI, which the API rejects — one
dead link swapped for another, plus the loss of the namespace signal.
Real per-asset RWA pages are separate work.
- A server response field shipped invisible to every consumer, and
two reconciliation gates could not see it (wave-D F-SDK-04). F-1321
moved the issuer's SEP-1 rounding hint off
decimals — where it
inflated market_cap_usd by up to 10^(7−display_decimals)× and was an
issuer-controlled manipulation vector — onto a new display_decimals
field. That field never entered the OpenAPI spec, so pkg/client and
the explorer's generated types both dropped it: the remediation's
entire replacement surface was unreachable from the published
product. A wallet had no way to obtain the issuer's stated
preference.
It is now documented in the spec, carried by the SDK, and present in
the generated explorer types.
Neither existing gate could have caught it. lint-docs compares
handlers to the spec at *route* granularity, never fields; the SDK's
contract test compares the SDK to the spec bidirectionally — which is
useful, but reconciles two *derived* artifacts, so when a field exists
only on the server they agree with each other about being wrong. A new
test compares the handler struct — the source of truth — to the
spec, fails on any undocumented response field, and fails loudly if
its own subject set comes back empty.
- Three SDK docs taught things the server does not do (wave-D
F-SDK-01/02/03).
TradeRow.BaseDecimals/QuoteDecimals were documented as a property
of the *asset* ("7 for native/classic/fiat"). They are a property of
the row's source, and a single page mixes both: on-chain rows
carry the asset's own scale, while CEX rows carry 8 regardless of
the pair. That error is payload-undetectable — price is
quote/base and therefore scale-invariant, so nothing in the response
looks wrong — and a reader assuming a constant mis-scaled real rows in
production once already. Corrected in the SDK, the handler, and the
OpenAPI description, since an SDK-only edit would leave the
machine-readable contract still teaching it.
MarketsOptions.OrderBy named the wrong server default: it said
alphabetic, but the default switched to volume-desc in May because
alphabetic surfaced spam tokens at the top. Corrected in all three
places the claim lived, and the godoc now says why a full-catalogue
walker should pass pair explicitly — volume-desc ranks on a
*mutable* key, so a pair whose volume changes mid-walk can be seen
twice or missed.
The SDK deliberately does not start sending order_by=pair: that
would silently flip every existing caller from top-by-volume to
alphabetically-first spam tokens, and make the SDK disagree with an
equivalent curl. Converting a wrong comment into wrong behaviour is
not a fix. Likewise rejected: widening parseAPIError to keep 256
bytes of an unrecognised body, which would let a proxy inject
arbitrary text into an error string that lands in customer logs.
- The ClickHouse lake's only self-healer was installed by nothing
(wave-D LID-1).
ch-live-catchup fills holes in the Tier-1 lake, and
both the script and its systemd units already existed and were
covered by the ops-credential test — but no task in configs/ansible
ever installed them, so on a playbook-provisioned host the healer was
simply absent. That is not cosmetic: resolveTip clamps to the
contiguous watermark, so a single unhealed gap stalls the CH-fed
projector permanently rather than degrading. It is reachable wherever
the lake is live — the testnet and futurenet inventories set
run_clickhouse: true, and the config template defaults both the live
sink and the projector source to true.
The role now installs the script and both units and enables the timer,
behind ch_live_catchup_enabled (default true, and only where the
ClickHouse tasks run at all). Units are copied verbatim from
deploy/systemd/ rather than templated: they carry no host-specific
values, and copying keeps the checked-in file the thing that actually
runs.
Reviewed, no change
- MSP-03 (four more surfaces bypass both gates:
/v1/markets and
/v1/pools last_price, /v1/chart?price_type=market_cap, and
windowed /v1/price). Two of the four are an open OWNER decision,
not a broken guard: last_price is documented as the raw
quote-per-base ratio — the same data class as deliberately-ungated
/v1/ohlc — and issue #366 states the unresolved scope question
verbatim. The windowed tier is additionally unreachable in the
shipped config, whose aggregate pair set is all crypto:/fiat:,
for which the scam gate returns false immediately. The root cause is
already recorded in #366 and #182.
- ALERT-05 (Alertmanager's inhibit rule keys on
component alone,
so one page suppresses every ticket of that component). The mechanism
is real and was reproduced against the shipped config, but the harm
claim is not: suppressed tickets are re-delivered ~5 minutes
after the page resolves, not at the next 24h repeat_interval —
inhibition is applied in the notify pipeline after the dispatcher
flush, so nothing is written to the nflog and the first post-unmute
flush notifies immediately. 14 of the 109 are informational alerts
routed to a receiver with no integrations, so they reach Discord
neither way; and every suppressed alert stays queryable with
inhibitedBy. It is also a duplicate of open audit finding OBS-02,
which records the same title, files and proposed fix. Left for that
finding's owner — an Alertmanager routing change is a production
paging decision — with the measured 5-minute number noted here,
because it materially lowers the severity OBS-02 was filed at.
- The R2/R3 deferral rationale rested on three false cache claims
(wave-D PS-07).
multi-region-ha.md said /v1/price,
/v1/oracle/latest and /v1/ledgers/latest "all return
Cache-Control: no-store". None is true at HEAD: /v1/price returns
public, max-age=30, s-maxage=60 — the SAME switch case as
/v1/assets, which the entry contrasted it against — /v1/oracle/
returns max-age=60, s-maxage=300, and /v1/ledgers/latest is not a
route at all (latest binds {seq}, fails to parse, and 400s). The
policy is the original April 2026 one, four months older than the
text, so "the deployed binary was older" was never available as a
defence. The conclusion survives on the real numbers — a 30-60s edge
TTL is not a substitute for a regional origin, since a Singapore
consumer still pays full origin RTT on every miss — but the argument
now says so from facts, and the micro-cache experiment it proposes
reads as more attractive rather than less.
- PS-03 (restore-drill's ClickHouse stage is opt-in via
DRILL_CH_WINDOW, so the scheduled monthly drill never measures lake
re-derive throughput). Real, but the opt-in IS the shipped documented
design, the gap is disclosed in three docs, and it is already tracked
by open issue #343. The finding's one increment over #343 — that
a metric alone could never populate, because the stage is gated — is
worth recording there, not re-filing. - PS-04 (ADR-0043 §2.3's "tail insurance" rests on a premise the
repo's own analysis contradicts, and is unimplemented). The premise
really is wrong-as-written, but ADRs are immutable
(
docs/adr/README.md) — superseded, not edited — and the corrected
assessment already lives in off-site-backup-plan.md with a drafted
amendment. Nothing to change without a superseding ADR, which is an
owner decision. - PFR-01 (the supply-divergence alert is unarmed on r1 because
[divergence.supply] is never rendered). Confirmed end to end, and
already stated in the alert rule's own comment plus a registered open
finding. Arming it is an operator/config decision on a production
paging surface, not a repo fix. - PFR-03 (a narrowed re-run can rewrite
tip_ledger downward). The
mechanism reproduces, but the defect CS-083 closed was AUTONOMOUS —
a nightly chunk driver that no longer exists. The trigger now needs
two deliberate operator commands, the first of which must find a real
problem; the end state is detected and annotated on the serving path
(coverageVerdictsStale, and the scenario's regression is 17× that
bound), CI-linted, and pinned by a test using materially identical
numbers. It is a re-report of CS-090's accepted residual. - PFR-05 (a blocked completeness write is a silent no-op). The
discarded
sql.Result is real; every consequence drawn from it is
wrong for the shipped configuration. The trigger is unreachable on
the deployed path (the driver's tip comes from a strictly monotonic
cursor), the claimed "fresh green verdict" prints complete=false in
the only deployed mode, and the backstop claim fails on all three
counts — a 36h per-source staleness gauge alerts on exactly the
column a discarded write leaves unchanged, naming the source, ~2h
after the blocked run rather than a day later.
- The capacity register offered two levers that no longer exist
(wave-D PS-05 / PS-06), on a document whose whole purpose is to be
read during a capacity crunch.
Move D (cold-tier enable + bulk LCM trim) was listed as an
unexecuted
~3.5 TB option whose AWS dependency was "not yet
incurred". It executed on 2026-07-26 and reclaimed 1.07 TB —
the estimate was ~3.5× high for a structural reason worth keeping:
early history is *sparse*, so trimming 78% of the partitions
reclaimed 22% of the estimate; the bytes live in the dense Soroban
era above the cutoff, which was kept. The dependency it was weighed
against is not just incurred but formally accepted (ADR-0043 §2), so
"adds an external dependency" no longer discriminates between the
remaining options. A planner would have added ~3.5 TB of
already-spent runway and ruled the option out on a criterion that no
longer applies. Move G's derived "~4 TB net" and the May-2026
recommendation table are corrected and annotated accordingly.
Move E (trades retention) read "Decision status: Lever
available" in a register that marks its dead levers explicitly.
Trades retention is forbidden: migration 0031 removed it, 0031's
own .down.sql names re-adding one as "the EXACT mechanism of the
recurring 'rogue retention on trades' data-loss drift", CLAUDE.md
carries it as a standing invariant, Ash re-signed it as launch
decision D5, and test/integration/migrations_test.go pins it.
Arming it would also trip the completeness verifier immediately —
migration 0116 treats a rising MIN(ledger) on a reconcile target
as loss, unconditionally, "because NO reconcile target has a
retention policy". Marked NOT A LEVER rather than deleted, so a
future reader sees why it was rejected instead of re-proposing it.
Also corrected: the data/postgres row still cited "ADR-0006
retention", which ADR-0006 itself records as superseded by 0031.
- The launch plan told an operator to mint a credential nothing
reads (wave-D PS-02). W4.5, W4.6 and Recommended-order step 3 all
carried the Go
sla-probe stack as live code with pending r1-ops
actions — "mint the Partner/Operator-tier key, set
stellarindex_probe_api_key in the r1 vault" — and cited
10-observability.yml line ranges as *installing* units that the same
file now *removes*. The whole stack was retired on 2026-08-24
(634d4be6, #135): both stacks wrote the same textfile, so the
keyless Go stack's 401/429 runs stomped the wrapper's passing
verdicts. An operator working the plan would have minted a live
operator-tier key with no consumer, whose file the next
--tags observability apply deletes — credential sprawl on the exact
surface W6.3 exists to shrink — then hunted a unit file ansible had
already removed. The genuine item, rotating the key exposed in a
2026-08-15 transcript, is preserved and now points at the file that
actually holds it.
- The launch-day migration gate named a version 7 behind HEAD
(wave-D PS-01). §2.8 hardcoded
schema_migrations.version = 143 when
head was 0150. Made version-agnostic rather than re-pinned — "143 →
150" just reproduces the defect at 0151 — pointing instead at the two
places CI keeps in agreement (migrations/ head and
ExpectedSchemaVersion, guarded by
TestExpectedSchemaVersionMatchesMigrationsHead). The gate's floor
semantics (applied >= expected, deliberately not ==) are now
stated, since the old text's "≤142 or dirty" phrasing invited reading
a schema AHEAD of the binary as a failure.
migrations/README.md's register was also missing rows for 0138-0143,
0145-0147, 0149 and 0150, against the file's own mandate. Backfilled,
and lint-migrations.sh gained a third pass that fails on a migration
with no row, a row naming no migration, or its own pattern going
vacuous. The reader this costs is the one the register exists for —
someone bringing up a fresh database, for whom the row is where an
"⚠ operator must re-materialize" warning lives. 0147 is exactly that:
it leaves nine CAGGs empty, and r1 having already run it does nothing
for a new node.
- `api.status_services` was validated case-insensitively and consumed
case-sensitively (wave-D RD-05). Config validation lower-cased each
entry before checking it against
{indexer, aggregator}, but
statusServicesOr only trimmed — and the heartbeat map is keyed by
Prometheus job labels with the stellarindex- prefix stripped,
which are always lower-case. So status_services = ["Indexer"]
booted clean and then reported "status": "unknown" on every
/v1/status request forever: overall never left degraded and the
explorer's status page stayed amber, while the operator debugging it
found a value that passed validation and matched the documented
vocabulary — the exact symptom the list was added (#328) to remove.
Both halves now apply the same transform.
- `NetworkUnavailable` promised to self-suppress and never did
(wave-D RD-06). Its docstring said it "renders nothing when the route
IS available here" and pointed at an
available helper "below" that
was never written; the component always rendered the empty state.
Nothing was visibly broken — all five callers guard with
if (!routeAvailable(…)) first — but the next network-gated surface
written by following that comment would have shipped "Not available
on Mainnet" above its real content, on mainnet (/exchanges and
/bridges are both in ROUTE_CAPABILITY with no page-level gate).
The component now honours the contract, and the available-route
branch — the case no test covered, which is how the drift went
unnoticed — is now covered. - The orphan-branch tripwire was about to report 17 non-problems on
its first real fire (wave-D RD-04). This repo's remediation flow is
a worktree fixer pushing
fix/issue-<N>, then a BATCH PR squashing
the verified subset (#353, #364) — and a squash-merge leaves no
ancestry, so a landed fix branch is mechanically indistinguishable
from a forgotten one: no PR, stale against main. On the first tick
where they cleared the 24h grace, every one of the 17 surviving
fix/issue-* branches would have been listed, all already landed
with their issues closed. A 17-row table of non-problems on a
tripwire's first real fire is how a tripwire gets ignored forever.
A fix/issue-<N> branch whose issue N is CLOSED is now treated as
*dispositioned* and kept out of the table — closing the issue is a
human act saying the work was dealt with, which is exactly the signal
this workflow exists to detect the absence of. They are still counted
in a footer naming them as safe to delete, because they are real
clutter, just not lost work; silence would trade one failure mode for
another. The rule applies only to that naming convention, and any
lookup failure falls through to REPORTING the branch — the tripwire
errs toward surfacing work, never toward hiding it.
Deliberately not done: loosening the 24h grace (it exists so ordinary
in-session branches don't spam the issue), and closing issue #282 —
that one is a live, unfixed P1 gap on main, and closing it would hide
a real defect. - The projector's replay-window read ran on the un-timeout'd root
context (wave-D RD-08). Every other
p.store call in the package
runs under cycleCtx; this one passed Run's ctx straight through,
so a query blocked behind a lock wait parked the watcher goroutine
with stellarindex_projector_replay_window_active holding whatever
it last published — a stale 1 keeps suppressing the lag ticket for
a source nobody is replaying, and the suppression's whole
justification is that it stays narrow. Not unbounded even before
(OpenBackground SETs statement_timeout on every connection, 30m
by default), but 30 minutes of a wrongly-suppressing gauge is not a
bound worth relying on when a local one costs two lines. Bounded by
PerSourceTimeout (60s) — deliberately NOT a budget matched to the
refresh interval, which would trip on ordinary DB slowness, zero the
gauge mid-replay and re-arm projector_lag_high for the whole
catch-up, reinstating the ticket storm #325 removed.
- RD-09 (replay-window upper bound on a widened dirty row). The
arithmetic reproduces, but the scenario is a *recorded, ratified
decision* —
docs/operations/runbooks/projector-replay.md documents
it with its operator remedy. Every proposed remedy is worse than the
gap: narrowing the range union, or refusing to record while a window
is pending, trades a tighter alert suppression for a data-integrity
regression on the verifier path — that union is what closed the
2026-07-31 carried-claim invalidation gap (19,366 over-projected
cctp rows), and compute-completeness's forced re-reconcile floor, the
table's primary consumer, depends on it. What was genuinely wrong was
a code comment: it glossed the bound as the pre-rewind position
unconditionally, which holds only for an un-widened row. Corrected,
along with a note on why the union must not be narrowed.
- `/v1/assets` accepted `order_by` and never read it, so the home
page's headline ranking was computed over the wrong ten assets
(wave-D RD-02). The explorer requested
?limit=10&order_by=volume_24h_usd_desc under the caption "Ranked by
trailing-24h trading volume across every venue we ingest"; the
handler built ListAssetsOptions with no Order, so it was served
the top ten by all-time observation count and re-sorted just
those ten client-side. An asset that traded $2M in the last 24h but
has a modest lifetime count could not enter the candidate set at all,
while a dormant high-lifetime-count asset held a slot and rendered as
a dash. order_by=TOTAL_GARBAGE returned 200.
This was missing WIRING, not a missing feature: the storage layer has
supported AssetsOrderVolume24hUSDDesc the whole time — its own
ORDER BY branch, keyset cursor args, cursor predicate and rank-tier
expression, and the unified path already passes it. The handler now
parses order_by, threads it into the query and both cursor
calls (the two orders encode different keyset keys, so encoding
under the wrong one skips or repeats rows rather than erroring), and
400s on an unrecognised value the way /v1/markets always has.
Combining order_by with asset_class now 400s rather than being
silently ignored: those listings rank on their own fixed scheme with
a cursor encoding that scheme's keys. Of the explorer's four callers
only the home page sends order_by, and it sends no asset_class.
The explorer's client-side re-sort is removed in the same change —
with the server ordering correctly it stopped being a no-op and
became actively wrong, because the API ranks on a concentration-
ADJUSTED volume (so wash / operational assets don't sit atop the
directory) while the payload's volume_24h_usd is the RAW figure.
Re-sorting the page by the raw column promotes exactly the assets the
server demoted. Native XLM, which /v1/assets does not return, is
now spliced into the server's order instead of triggering a re-rank
of everything.
- `pkg/client` SDK: `Retry-After` no longer yields a NEGATIVE
back-off.
parseRetryAfter multiplied the header's delta-seconds
into a time.Duration (int64 NANOSECONDS) with no range check, so
any value above ~292 years wrapped — Retry-After: 9223372036854775807
produced -1s, and a caller sleeping on it retried IMMEDIATELY,
the exact opposite of the back-off requested. Out-of-range values
now return 0, the field's already-documented absent/unparseable
sentinel. Deliberately not a clamp: APIError.RetryAfter is a
SemVer-stable *reporting* field, and clamping would make it
misreport the wire (wave-D F-SDK-06). - `pkg/client` SDK: an oversized response body now errors instead
of surfacing as a bogus JSON decode failure. The 16 MiB read cap
used
io.LimitReader at exactly the limit, and LimitReader
returns (n, nil) AT its limit — so a truncated body was
indistinguishable from a complete one and got parsed, reporting a
confusing error about the payload rather than the truth. Now reads
cap+1 and errors naming the cap (wave-D F-SDK-08). - Docs: `pkg/client` query-parameter godoc said out-of-range
`limit` / `window_seconds` values are "clamped". They are
REJECTED with a 400. Read in the
std::clamp sense the old wording
told a caller their out-of-range value would be quietly honoured at
the boundary — on a pricing surface, the difference between a VWAP
over a window they never asked for and a loud error. The
genuinely-saturating ADR-0015 closed-bucket adjustment keeps the
word, and docs/architecture/lexicon.md now fixes both meanings so
the two do not re-blur. ADR-0018's copy of the old wording is left
alone — ADRs are immutable (wave-D F-SDK-09). - **Docs: the documented
pkg/* SemVer release mechanism was
inert.** semver-policy.md and release-process.md instructed
cutting pkg/client/vX.Y.Z tags, but this repo is a single Go
module (ADR-0005), so such a tag versions nothing — the proxy has
no nested module and go get …/pkg/client@v0.2.0 fails outright.
Both documents now state that pkg/client ships on the root clock,
that a pkg/* break bumps the root minor and MUST be named in the
CHANGELOG (the consumer's only notice), and why adding
pkg/client/go.mod would be a live break for everyone currently
pinned on the root module rather than a fix (wave-D F-SDK-05,
#361 item 8).
- CV-2 (oracle reconcile netting). The finding reads an unwired
vintageBoundary field as a live hole; the history is the reverse.
It shipped and changed behaviour, then was retired because its only
subject was upgraded to a *stronger* position — strict per-ledger with
no netting, proven over 12.5M ledgers with zero mismatches. The
remaining oracle netting is a recorded, deliberately-deferred decision
(F6 / C2-16) with a register entry, a rationale and a superseding
design doc. The implied fix — setting a boundary on the four oracle
sources — is not actionable: no cutover ledger for the legacy backfill
exists anywhere in the repo, and guessing one too low re-opens the
false-positive class the deferral exists to avoid.
- CV-4 (recognition claim unfalsifiable for the sep41 sources). The
mechanical observation is right, but the remedy would be actively
harmful. Lifting the topic exclusion cannot create a falsifiable
check — every watched contract maps back to a sep41 source, so the
guaranteed gaps would pin
recognition_ok=false and therefore
complete=false *permanently*, while every non-watched SAC's shapes
flood the unattributed bucket. The exclusion's justification is also a
structural truth about the dispatcher this function builds, not the
stale deployment observation the finding assumes. - Two migration headers cite a different migration than the file they
are in (wave-D CV-7).
0125_projection_dirty_windows.up.sql opens
with 0124 up and 0096_create_blend_emitter_events.up.sql with
0095 up — both real but unrelated migrations, so a reader following
the reference lands somewhere else. A Go comment describing the
dirty-window table cited migration 0124 for the same reason; that
one is corrected.
The two migration headers are not, and cannot be: applied
migrations are immutable, and even a comment-only edit changes the
checksum. Immutability is the stronger rule — a migration whose bytes
can change is one whose applied-ness cannot be proven — so the drift
is recorded rather than fixed, and a new lint stops the set growing by
failing any *new* migration whose header cites the wrong number,
before it ships and freezes.
A third citation, in freeze_events.go, says 0124 and is correct
(0124 really is freeze_reason_other) — a blanket find-and-replace
would have broken it. The check is therefore deliberately narrow: a
file disagreeing with its own filename, not whether every migration
NNNN mention in the tree points at the right subject.
Added
- Tests for the CS-017 price-freshness seams (wave-D PFR-04).
storePriceReader's now func() time.Time and vwapFreshness
fields exist only to be injected by a test, and nothing did — so the
15-minute staleness rule had no enforcement beyond runtime. Now
pinned: the default window and why it is 15 minutes, the zero-value
sentinel (an explicit 0 must mean "unset", not "never stale"), the
injected clock, and the staleness boundary mirroring LatestPrice's
real expression including its measure-from-CLOSE +1m and its
lowConfidence short-circuit.
PFR-04's *failure scenario* does not survive and was not acted on:
the dormant long tail cannot resume being served a months-old bucket,
because the substance gate runs twelve lines earlier, its window is
trailing-24h, and a dormant pair fails its first comparison — the
read returns ErrPriceWithheld and the staleness expression is never
evaluated. The end-to-end read also remains outside unit-test reach
(storePriceReader.s is a concrete *timescale.Store with no
injectable constructor); that belongs to the integration harness.
- Composite-reference corroboration of the phase-2 freeze for
structurally single-venue targets (product decision, Ash
2026-08-29; design doc §10.1 amendment). For an allow-listed target
(
[aggregate.composite_reference], default ON for
crypto:XLM/fiat:GBP + crypto:XLM/fiat:EUR) whose bucket is
single-venue, the aggregator rebuilds the target's triangulation
chain on the CURRENT bucket — this tick's crypto/USD leg publish
(≥ min_leg_sources real venues, default 2) × a fresh FX snap
(≤ fx_max_age_hours, default 76, FX source class only, never an
oracle) — and reads it against the fresh direct VWAP: agreement
within tolerance_bps (default 75) means the move is market-wide
and the 3-signal-AND fire is suppressed (corroboration_basis=
composite); disagreement or an unavailable reference freezes
exactly as before, the reason string naming why
(corroboration_basis=venue composite_unavailable: leg_sources=1
composite_leg_sources={…}). The same sample feeds the confidence
factor (triangulation_checked) and the mid-hold release lens, so
a corroborated genuine move can also release. Hard invariants: the
composite never enters VWAP and never raises source_count /
effectiveSourceCount; targets with ≥ 2 real venues are
byte-identical to before. New: composite_meta.corroboration_basis
+ composite_leg_sources, gauges
stellarindex_aggregator_composite_corroboration{pair,window,verdict}
/ ..._composite_reference_leg_sources{pair,window,leg}, counter
..._composite_freeze_suppressed_total. Never a prior tick's sample
(the rejected 95da898d mechanism). Tests:
TestCompositeReference_* (manipulation control with the mechanism
ON, market-wide mirror, stale-FX / thin-leg / oracle-FX fail-closed,
multi-venue differential, exact-Rat tolerance boundary, refresh
order) plus the unchanged
TestRouterFreeze_TwoRoutesSuppressSingleSourceFreeze 3-tick control
(#246). Verifier advisories (same day): A1 leg-dispersion guard —
every venue's own bucket VWAP on the crypto/USD leg must be within
leg_dispersion_bps (default = tolerance_bps) of the leg VWAP,
else composite_unavailable: leg_dispersion=… (two venues only count
as two when they agree; gauge
stellarindex_aggregator_composite_reference_leg_dispersion_bps);
A2 the mid-hold release lens for a resolved reference uses a
dedicated release_band_pct (default 2.0), not the shared 5 %
cross-oracle band — a held +4 % venue-specific offset no longer
auto-releases (TestCompositeReference_ReleaseBandHoldsVenueOffset,
…_LegDispersionCannotCorroborate, …_LegDispersionBoundary,
TestLegDispersion_MeasuresWorstVenue). A3/A4: config-bound tests
(TestValidate_CompositeReferenceBounds) and the guard fails CLOSED
when a venue VWAP cannot be computed (leg_dispersion=uncomputable,
TestCompositeReference_UncomputableDispersionFailsClosed).
- Rolling ZFS snapshots of the ClickHouse lake + Postgres on r1
(decision 2026-08-29).
scripts/ops/zfs-snapshot.sh (installed by
the archival-node role, new tag zfs-snapshots, zfs-snapshot.timer
daily 01:45 UTC) takes auto-YYYYMMDD-HHMM snapshots of
data/clickhouse (3 d retention) and data/postgres (7 d) — the
minutes-scale answer to a logical fault (bad migration, DROP, bad
re-derive) alongside pgBackRest's hours-scale PITR. Hard min-free
guard (zfs_snapshot_min_free_bytes, default 2 TiB): below it the
job prunes its oldest auto-* snapshots (never a dataset's newest,
never any non-auto-* name) and, if still below, skips the snapshot
and reports stellarindex_zfs_snapshot_guard_skipped=1. Textfile
gauges (stellarindex_zfs_pool_free_bytes,
stellarindex_zfs_snapshot_{latest_unix,count,used_bytes}), alerts
in both rule trees (zfs-snapshots.yml: pool free < 2.5 TiB ticket
/ < 1.5 TiB page, snapshot > 36 h stale) with promtool tests, runbook
docs/operations/runbooks/zfs-snapshots.md (honest crash-consistent
semantics for ClickHouse and Postgres, clone-and-copy / rollback
procedures, vs pgBackRest PITR), and
scripts/ops/zfs-snapshot-now.sh <dataset> [--keep <label>] for the
fresh-snapshot precondition of the ClickHouse destructive-DDL
runbook. The guard is fail-closed: unreadable zpool free space
(command failure / non-number) aborts the run before any destroy or
snapshot, exits non-zero and emits
stellarindex_zfs_snapshot_pool_free_unreadable=1 (own ticket).
Invariants pinned red-first by scripts/ci/zfs-snapshot-test.sh
against a stubbed zfs, including the destroy choke point directly. - No-orphan-work contract + daily `orphan-branches` tripwire. On
2026-08-27
fix/priceless-structural-unpriceable was pushed with no
PR and no backlog line; on 2026-08-28 a different agent re-diagnosed
stellarindex_assets_popular_priceless from scratch and fixed it
differently (#254), and the orphan (plus a postmortem branch, now
#255) surfaced only via a manual branch audit. The contract is stated
once in AGENTS.md (push ⇒ PR same session; prior-art check via
gh pr list --state all --search, git branch -r, backlog + runbook
grep; PR names the alert and root cause vs symptom; supersede by
closing with a comment) and cross-referenced from CONTRIBUTING.md and
CLAUDE.md. The PR template gains Alert / finding and Prior
art fields. .github/workflows/orphan-branches.yml (daily +
workflow_dispatch, contents:read / pull-requests:read /
issues:write) lists every remote branch other than main /
old-* / archive* with no open-or-merged PR and a last commit
>24h old, and opens/updates a single "Orphan branches (no PR)" issue
(closes it when the list is empty). - `stellarindex_ingest_gap_detector_silent` third clause (both rule
trees). The alert's
absent_over_time(runs_total[15m]) clause is
satisfied by the outcome="error" counter, and a target that has never
once succeeded in a process life emits no last_success_unix stamp to
age — so a scan failing every cycle (the 2026-08-28 r1
soroban_events statement_timeout loop, found verifying #258) fired
nothing. The rule now also fires when the target's error counter is
present now and 8h ago with no last-success stamp seen in 8h. First
promtool unit tests for the alert
(deploy/monitoring/rule-tests/ingestion_test.yml): fresh stamp
silent, stale stamp fires, never-succeeded fires, stamp-within-8h
suppresses, aggregator-absent fires. - Ops scripts honour a ClickHouse ops user
(`scripts/ops/ch-ops-user-test.sh`).
ch-live-catchup.sh,
ch-supply-flows-seed.sh, d2-ordinal-reproject.sh,
d3-lecur-v2-rebuild.sh and ch-backfill-monitor.sh ran
clickhouse-client as the default user with no way to supply
credentials. They now honour STELLARINDEX_CLICKHOUSE_OPS_USER /
STELLARINDEX_CLICKHOUSE_OPS_PASSWORD (e.g. from
/etc/default/stellarindex-ops), handed to the client through its
CLICKHOUSE_USER / CLICKHOUSE_PASSWORD environment — never argv,
which ps and the journal would show. The monitor resolves them on
the HOST side of its ssh (new OPS_ENV, default
/etc/default/stellarindex-ops) for the same reason. Unset ⇒
byte-identical invocations; the new stub-backed test pins both the
credential hand-off and the unchanged argv per script, and runs from
scripts/dev/verify.sh. - AWS Public Blockchain dataset drift monitor (audit 2026-08-29,
backup-restore-6). r1's galexie-archive was trimmed below ledger
49,984,000 on 2026-07-26, so the second raw-LCM archive ADR-0043
relies on is the third-party
aws-public-blockchain pubnet dataset
— and nothing watched it. .github/workflows/public-dataset-check.yml
(weekly + dispatch, no credentials, --no-sign-request, first-party
actions only) now asserts contiguous 64,000-ledger coverage from
genesis to ≥ tip − 2 partitions, the HEX--start-end naming, an
unchanged .config.json manifest and the trimmed range
[64000, 49983999] fully present; drift opens/updates ONE "AWS
Public Blockchain dataset drift" issue (auto-closed when intact) and
never fails the scheduled run red. Decision core
scripts/ci/check-public-dataset.sh, fixture-tested in ci
(check-public-dataset-test.sh: gap inside/above the trimmed range,
misnamed partition, manifest change, stalled publication all RED).
Changed
- The two frozen planning inventories are retired, and
`verify-launch-ready` can no longer certify a retired document
(#321).
docs/architecture/launch-readiness-backlog.md had gained
zero rows since 2026-05-13 and contained none of the actual v1 gate
(W6.1 paging, W6.3 rotations, W4 backups, the W8 correctness
backlog, ToS/Privacy #237), yet every L1–L5 row still carried its
last-written ✅ — so the weekly launch-readiness.yml workflow
republished *"✓ Engineering surface ready"* over a document that had
stopped tracking reality, and got more confident the staler it got.
The doc is now formally superseded by docs/operations/v1-launch-plan.md
(frontmatter status: superseded + a banner mapping its still-open
rows: L4.14–L4.17 + L5.8 → W9 gated on D2, L5.6 → W6.2, L6.4 → §2.8,
L6.6/L6.7 → W6.7), .github/workflows/launch-readiness.yml is
deleted, and scripts/ci/verify-launch-ready now reads the
frontmatter and emits no verdict at all (new exit code 3) for any
document declaring itself superseded/retired — a retired doc's rows
are history, and neither a green nor a red computed from them means
anything. The prior 2026-07-24 staleness banner was itself wrong
twice over (it claimed the gate was unwired the day *after* it was
wired in 068ec709, and named a "current source of truth" that was
superseded on 2026-07-27); both corrections are recorded in the new
banner. docs/operations/open-fixes-inventory-2026-08-08.md is
superseded on the same terms: 24 of its 35 rows were done and never
struck — rows 1 and 19 closed on the day it was compiled
(d1cd18ac, a1c5c2e5), rows 2 and 5 two days later (f75ab4b2,
ef278218) — and its genuinely-open threads are carried into the
launch plan's §5. The public company page's "roadmap that gets us to
v1" link now points at v1-launch-plan.md instead of the retired
backlog. Tests: TestRealBacklog_IsRetired,
TestVerdictLine_RetiredDocNeverCertifiesReady,
TestSupersession_ReadsFrontmatterOnly, and a company-page case
pinning the roadmap href (red-proof: the pre-fix binary run against
the now-retired doc still printed "✓ Engineering surface ready
(subset gate)" and exited 0). - ADR-0043 §2 amended (2026-08-29): "two independent raw-LCM
archives" now explicitly = our recent range + the AWS Public
Blockchain dataset; dependency accepted and monitored rather than
duplicated, with the one-time cross-region copy (≈ $80 + $3–4/mo)
recorded as the not-taken option.
off-site-backup-plan.md status
carries the same note. - Public status page shows backup freshness (Ash, 2026-08-29). New
read-only
GET /v1/diagnostics/backups (experimental) reports the
pgBackRest last full / diff / WAL-archive age, the per-repository
newest backup (repo 1 on-host, repo 2 encrypted S3 off-site), the
monthly restore drill's last run + pass/fail, and the ClickHouse
schema+state snapshot age — each with a freshness verdict
(ok / stale / unknown) against SLOs the API echoes in slo
(full ≤ 8 d, diff ≤ 36 h, WAL ≤ 15 m, off-site ≤ 8 d, drill ≤ 35 d,
snapshot ≤ 36 h). Source of truth is Prometheus — the same
pgbackrest_exporter / node_exporter textfile series the alert rules
read; the API never shells out to pgbackrest. Every timestamp is
nullable and an absent series is null + unknown, never a fresh
zero; source_status carries the document's trust tri-state and
flags.stale mirrors the roll-up. Cached 60 s, no secrets or paths;
503 where no api.prometheus_url is configured. The explorer
/status page mounts a Backups panel (BackupsPanel.tsx,
mainnet only) that renders green within SLO, red with the real age
past it, grey "no data" for absent sources, and a "verdicts not
trustworthy" marker when the API's Prometheus reads failed — ages
come from the API's age_seconds, never the browser clock. Reserved
nulls (documented in the spec): repo retention, drill
restored_backup_ts / duration_s, zfs_snapshot_latest_ts,
replica_lag_s — no producer exports them yet. Tests:
internal/api/v1/diagnostics_backups_test.go,
web/explorer/src/app/status/BackupsPanel.test.tsx. - `stellarindex_backup_offsite_stale` (P3, both rule trees). The
existing backup alerts read
pgbackrest_backup_since_last_completion_seconds,
which the exporter computes ACROSS repos — a host whose on-host repo1
is fresh while every repo2 (S3) write fails stayed green and the one
copy that survives host loss aged out silently. The new rule fires
per UP exporter instance with no pgbackrest_backup_info{repo_key="2"}
series younger than 8 d (x unless x offset 8d — a new backup is a
new series), which also covers repo2 never written. promtool tests
in deploy/monitoring/rule-tests/backup-offsite_test.yml (red-proof:
widening repo_key to all repos makes the repo1-fresh/repo2-stale
case stop firing); runbook runbooks/backup-offsite-stale.md.
- `/v1/assets` now rejects a malformed catalogue cursor instead of
silently serving page 1 (wave-D KP-4).
catalogue:abc,
catalogue:-7 and an Atoi-overflow were swallowed and treated as
"no cursor", while every sibling paginated surface 400s on the same
input. This is a wire-behaviour change (200 → 400), though unreachable
through any shipped client: the explorer clamps limit to
{50,100,200,500} and a catalogue: cursor is only emitted below
limit 11. - `/v1/assets` declared `limit` twice (wave-D KP-5) — an inline
parameter with no default, alongside
$ref: Limit which defaults to
100. Generators pick one arbitrarily, so the rendered reference, the
Postman collection and the explorer's generated types could each
disagree about the same field. The inline copy is removed and its one
unique fact (page 1 fills from the classic stream when the catalogue
is shorter than the limit) folded into the operation description; all
three spec-derived artifacts are regenerated.
Spectral *does* flag this, as operation-parameters at severity
warn — CI simply runs the action at its default
--fail-severity=error, so it never failed the build. lint-docs now
enforces resolved-parameter uniqueness as a hard gate, which avoids
re-tuning Spectral's global severity floor and lighting up unrelated
warnings.
- Follow-ups from the adversarial review of the wave-D merges
(2026-08-31 sweep — the merges were self-reviewed, so this pass
existed to catch what that misses; it did).
stellarindex_oracle_stale's join used on (source), which discards
job/instance from the match. Two scrape targets exporting the same
source make the right side non-unique and the rule fails evaluation
outright — *"found duplicate series for the match group … many-to-many
matching not allowed"* — which is the pre-fix silence plus noise. The
prometheus job template builds stellarindex_indexer by looping its
host group, so a second indexer host (R2, ADR-0004/0016) produces
exactly that shape. Now ignoring(asset) group_left(), with a
two-target regression case.
The price-withholding seam guard advertised that "a new read seam
cannot forget it" while enumerating two hard-coded seam names — so
a brand-new ungated reader passed it silently, and the property it
claimed did not exist. It now derives its subject set by finding every
method that calls a closed-VWAP store read (a reader must call one to
serve a price at all), with two documented exemptions and a fatal on an
empty subject set. Two sibling guards carried the same overclaim: the
keyset-ordering test now enumerates the orderings, and the explorer
truncation guard's scope is corrected in its comment rather than
widened — proving that class repo-wide needs dataflow analysis, and a
widened regex flags correct code.
withholdPriceSeriesWhenUnpriced also nulls ath for declared-peg
assets, not only scam-flagged ones. That is intended — GetAssetATH
reads the asset's own USD-quoted market, which for a declared-peg
asset is the dust market the substance gate refused, while the
published headline comes from the peg — but it shipped unpinned and
undescribed. Now pinned by a test that says why.
The runbook write-gate lint matched flagset names with [a-z0-9-]+,
which cannot match a space, so five two-word write-gated
subcommands (supply snapshot, supply seed-observations,
supply seed-sac-balances, supply seed-claimable-balances,
supply seed-sep41-genesis) were silently absent from the gated set: a
runbook telling a responder to run one without -write produced no
finding, and the command reports errors=0 having written nothing.
Widening it surfaced a false positive on a deliberate -dry-run
preview, so explicit -dry-run is now exempt — flagging those would
train responders to ignore the check.
- ~20 systemd oneshot timers had no failure alert, including the sole
writer of the table the scam-pricing gate reads (wave-D LID-6).
directory-sync populates account_directory, which the gate
consults on every aggregated price serve. It is Type=oneshot with no
OnFailure and emits no metric — so if it stopped, the table froze at
its last good snapshot, a newly-flagged scam issuer was never learned,
and the gate kept serving. That reproduces the incident the gate was
built to stop, with the gate present, correct, and reading stale
input.
Naming units individually is how the gap opened, so
stellarindex_systemd_unit_failed inverts it: every unit is covered
unless it has a dedicated alert with better triage. The five
exclusions live in scripts/ci/unit-failed-dedicated.baseline, and a
self-test asserts each is genuinely named in a rule file — an
exclusion cannot become a silent suppression, and an empty baseline
fails rather than passing vacuously.
The runbook leads with the property that makes these failures hard to
spot: these units write data something else *reads and then trusts*,
so a failed sync surfaces as a consumer serving stale data
confidently, elsewhere, possibly days later — not as an error.
- Corrected the `/v1/price/batch` fan-out comment (wave-D
UNAUTH-DOS-2). It claimed 16-wide parallelism stays "well inside the
DB connection pool's headroom even with several batches in flight",
which does not hold for a 1000-id POST batch. The constant itself is
unchanged and should not be lowered as a throughput control: narrowing
it removes zero database work while lengthening how long each request
holds its connections, re-creating the regression it was raised to
fix. The bound that matters is the rate limiter, and charging it per
id rather than per request is the sound way to close the
amplification.
- An unauthenticated client could leave unbounded price-tip compute
loops running (wave-D UNAUTH-DOS-1). The SSE caps count
*connections*, but a
/v1/price/tip/stream connection also mints a
detached producer: its context comes from context.Background(),
it outlives the request by design, and it survives the connection's
release for a 30-second linger. So opening and immediately aborting
streams in a loop left a growing set of compute loops, each polling
the database on its own ticker, with no connection left for the
connection cap to see. The Hub's topic reaper could not shed them
either — it evicts only subscriber-less topics, and a live producer
recreates its topic every window.
Cheap to drive, because the producer key includes a client-chosen
window_seconds in [1,60]: the key space is pairs × 60, so an
attacker needs no distinct assets at all. The regression test
enumerates exactly that, and pre-fix left 32 producers running from a
single pair.
Distinct producers are now capped (default 512, SetMaxTipProducers
to tune, negative to disable). Two deliberate properties: a pair that
already has a producer still admits new subscribers at the ceiling
— those cost nothing extra to serve, and refusing them would penalise
a popular pair's own audience — and a refused stream returns
503 + Retry-After rather than falling through to the legacy
per-connection loop, which is the unbounded compute the ceiling
exists to prevent.
- The `/assets` "#" column restarted at 1 on every cursor page
(wave-D EXR-06), so the 101st asset was labelled
#1 under a header
that reads as a global rank. The counter is per-page, and cursor
pagination keeps only the opaque cursor in the URL — there is no page
depth to recover. The rank is now shown only on the unpaginated first
page.
Deliberately suppression rather than arithmetic: deriving
depth * limit + i would print a *different* wrong number, because
suppressCatalogueTwins and foldAliasTwins drop rows after the
query so pages under-fill (measured 81/96/99/96 at limit=100). A
rank the data cannot back is better omitted than guessed.
The test mock hardcoded an empty query string, so every existing case
ran on page 1 — which is why nothing caught this. It is now settable,
and a case pins the paged behaviour.
- Every asset link pointed at the bare CODE, so a link could resolve
to a different issuer's asset than the row clicked (wave-D EXR-02).
assetSlug truncated a canonical CODE-GISSUER… id at the dash, and
/assets/USDC is shared by every USDC-alike — so a link built from a
scam issuer's row could land on the legitimate asset's page, or the
reverse. AssetLink and AssetText now link the full canonical id,
as /markets/[pair] already did for the same reason (AM-09).
The docstring justifying the truncation ("long-form ids are NOT in
generateStaticParams … so linking to them hard-404s") had outlived its
constraint: canonical asset_id routes are emitted for exactly the
same asset set as the short slugs, and anything outside that set falls
to the client shell under both spellings — so the canonical form never
links worse and always links precisely.
Display labels are unchanged: shortAssetText stays short, because
these are dense analytics rows and a 56-char id would blow out every
cell and chart legend. AssetText carries the canonical id in title
instead, so the issuer is recoverable on hover without spending row
width.
Guarded three ways: the repo-walk pack gains a rule against building
an /assets/ href from a code-truncated id, and assetSlug — which
decides where every asset reference in the explorer points, and had no
behavioural test at all — now has one, including the property that two
issuers sharing a code get different links. All proven red pre-fix.
- A scam-flagged asset outside the pre-rendered top 500 showed no scam
warning at all (wave-D EXR-01).
/assets/[slug] has two render
paths: the build-time pre-render for the top 500, and a client shell
(AssetPathView) for everything else. Both fetch the same
/v1/assets/{id} payload, carrying the same issuer_directory_tags
and issuer_scam_reason — but the banner ("Do not trust this asset,
establish trustlines, or execute the prices below…") was inlined in
the pre-rendered page only, and the shell simply ignored those fields.
The path serving the long tail, which is where a scam token actually
sits, was the one rendering without the warning. The banner is now one
shared AssetScamCallout mounted by both paths.
Also closes EXR-05 / part of #335: AssetSwap's TokenIcon
rendered an issuer-controlled image URL without the SEC-10
isSafePublicImageUrl host check that the other two <img> sites
apply.
Both were the same failure mode — a trust-critical rendering
obligation enforced by convention at some call sites and silently
absent at one — so both are now pinned by a new guard pack,
src/lib/trust-surface-guards.test.ts, which enumerates the call
sites from src/ at test time. A fourth <img> site or a third asset
view fails on the day it is written. Both guards were verified red
against the pre-fix tree, each naming its offending file.
- One extra path segment defeated the price-withholding gates
(wave-D MSP-01).
/v1/price correctly returned
errors/price-withheld for a directory-flagged scam issuer or a
market too thin to aggregate — while /v1/price/at and
/v1/price/changes published the identical closed-bucket VWAP,
because storePriceAtReader carried neither gate. Every
price-serving read seam now routes through a single chokepoint,
priceWithheld(), which is the only place in the binary where either
gate is spelled.
The same change closes MSP-07: the last-trade arm of
LatestPrice consulted the thin-market gate but not the scam gate,
so an operator setting pricing_guard.disable_substance_gate=true to
diagnose a pricing-coverage complaint silently also un-withheld every
directory-flagged issuer's last trade — reversing a separate,
owner-level trust decision they never touched.
Two AST guards pin this, both proven red against the pre-fix state:
one enumerates the price-serving seams and fails when a seam does not
route through the chokepoint; the other fails on any
substance.Allowed/scam.Withheld call outside it. The second is
the one that catches MSP-07 — a seam can call the chokepoint on one
arm and still hand-roll half the decision on another.
Deliberately unchanged: /v1/twap, /v1/vwap, /v1/ohlc and
/v1/chart remain ungated. The raw-trade surfaces are documented as
deliberately visible (pricingguard/scam.go), and which quote sets
constitute a "price claim" is the open scope question in #366. - `GET /v1/assets` silently truncated its own pagination on any tie
(wave-D KP-1 / RD-01 — the same bug found twice, independently). The
keyset cursor predicate for the default observation-count ordering
compared
(observation_count, asset_id) < ($n, $m) — a SQL row
constructor, which compares every element in the *same* direction —
against an ORDER BY observation_count DESC, ca.asset_id ASC, which is
mixed-direction. On a tie in observation_count the tie-break half
therefore read as asset_id < $m, re-selecting rows the walk had
already served while skipping the ones it had not. A client paging the
full asset list received some assets twice, never received others, and
was then told has_more: false as though the walk had completed. Ties
are the norm in the long tail, where most assets share a small
observation count. The volume ordering always spelled the comparison
out correctly; only this arm was wrong.
The existing pagination regression test could not catch it: its fixture
gives every row a distinct observation count *and* a distinct volume,
so the walk never crossed a tie. It now also seeds rows that tie on
both sort keys. A source-derived unit invariant additionally asserts
that any ordering whose ORDER BY breaks ties on asset_id ASC
resumes with asset_id > $n and uses no row constructor — so a third
ordering added later is covered on the day it is written, rather than
when someone notices missing rows.
- `make verify` was red on `main`, and CI could not see it. The
ClickHouse ops-credential contract (
scripts/ops/ch-ops-user-test.sh)
has been failing since #286 gave d2-ordinal-reproject.sh a
destructive-DDL acknowledgement (D2_FORCE_DROP) that exits *before*
the script's first query — so the harness never got a stubbed
clickhouse-client invocation to assert on, and reported
clickhouse-client was never invoked. The canonical pre-push gate has
therefore been failing for every contributor who ran it. The harness
now acknowledges the guard (safe: the stub fails the first, read-only,
query, so the script bails long before any REPLACE PARTITION or
DROP, and CH_FLAGS_DIR is redirected away from the real flags
directory). Root cause of the *silence*, now also fixed: this contract
ran only in scripts/dev/verify.sh and in no CI job, so it shipped
red — it is now wired into the ops self-test step alongside the
restore-drill contracts. 15/15 passing.
- Runbook re-verification wave K: eight alert runbooks re-derived
against HEAD (7 broken, 1 stale), plus a lint so the worst class
cannot recur.
ledgerstream-tier-both-missing.md — a P1 page —
told responders to "pull the missing range from R2 or R3's mirror"
with rehydrate-galexie-archive … --source vultr. There is no peer
selector and never has been: the command only ever reads the
CONFIGURED cold tier (storage.s3_cold_*), so during an AWS Open
Data outage — the exact scenario the step sat under — it cannot
route around anything. Its -write fail-closed note was correct
(the shared opsutil write gate makes dry run the default) and is
kept, now with the trap that motivated the re-verification spelled
out: a dry run buckets every not-in-hot path as copied WITHOUT
asking cold whether it holds the object, so a forgotten -write
logs copied=N missing_in_cold=0 errors=0 and exits 0 — a
success-shaped report having rehydrated nothing. It also cited two
gauges that have never been registered.
postgres-ping-failing.md still documented the > 0.5/s threshold
that was unreachable by 30× at the 60 s probe cadence (corrected to
> 0 in both rule trees on 2026-08-04) and a
trade_inserts_total{outcome="error"} label that does not exist.
source-stopped.md described ONE 30 m × 15 m alert; the shipped
shape is the F-1208 three-way split (high-volume 30 m/15 m,
low-volume DEX 24 h/30 m, daily publisher 30 h/1 h) all sharing one
runbook_url. external-poller-stale.md blanket-claimed "30
minutes", misdescribing the 12 h ECB rule by 24×.
ingestion-duplicate-flood.md still said ON CONFLICT DO NOTHING
(INV-3 / migration 0109 made it a generation-guarded DO UPDATE, so
a corrective re-derive now reproduces the alert's exact signature)
and used -sources, which does not parse (-source, singular,
comma-separated).
decode-errors.md asserted a pre-P23 operations+effects fallback
that has never existed, a pre-ADR-0035/0040 comet topic-only match,
and a redstone length-mismatch that refuses without recovery.
sev-status-page-update.md still pointed at web/status/, now a
redirect-only stub — the live page is the explorer's
web/explorer/src/app/status/ (CLAUDE.md's copy of the same drift
is fixed by #326, above). exporter-down.md
attributed r1's exporters to the redis-sentinel role, which r1 never
runs. Guard: lint-docs.sh §11's runbook metric-name check was
scoped to stellarindex_source_* only, so both phantom gauges above
were invisible to CI; it now covers every obs-owned namespace a
runbook cites (source/cursor/indexer/backfill/trade/postgres_ping)
and resolves histogram _bucket/_sum/_count children (ansible
inventory variables sharing the namespace are subtracted, derived
from configs/ansible/, so a runbook naming
stellarindex_backfill_from_ledger can't red CI as a false
positive). A second guard, §11b, fails CI on any
run-heavy-job.sh … stellarindex-ops <sub> invocation in a runbook
where <sub> registers the shared write gate but the command omits
-write — the wrapper is the COMMIT path, so a dry run under it is
always a bug. The gated-subcommand set is derived from the Go
source. Both checks verified red against the pre-fix runbook text. - The P1 archive-divergence page can actually fire: verify-archive now
exports its mismatch counter through node_exporter (#282).
stellarindex_stellar_archive_divergence (severity page) selects
stellarindex_verify_archive_mismatches_total, which the chain /
checkpoint walk increments — but the counter's only export path was the
opt-in -metrics-listen HTTP endpoint, which neither
verify-archive-tier-a nor -tier-b passed and
configs/prometheus/prometheus.r1.yml has no scrape job for (a one-shot
job is gone between scrapes anyway). The metric had no producer in the
deployed topology, so a genuine archive-correctness event opened a P3
ticket (stellarindex_verify_archive_unit_failed) and paged nobody; the
2026-06-11 F-1329 repoint had fixed the metric NAME but not the export
path. New -textfile-output PATH flag writes the counter into
node_exporter's textfile_collector dir, wired into both units (ansible
templates + the deploy/systemd reference copies) with per-unit .prom
files. Three properties make it usable by increase(): totals are
CUMULATIVE across runs (a clean run re-emits, never resets), all three
reason values are ZERO-SEEDED on every run (a series that first appears
at 1 and stays flat yields increase() == 0 — the same F-0033 /
C4-038 "absence reads as health" trap as the gap-detector fix below),
and the series is labelled by tier rather than chunk_idx (a per-run
worker slot with no cross-run meaning, and two units exposing an
identical label set through one node_exporter target is a duplicate-
metric scrape error). The rule's lookback also widened 1h → 26h:
against a NIGHTLY producer a 1h window showed the step for one hour in
twenty-four, so a SEV-1 correctness page self-resolved before the
morning. Pinned by deploy/monitoring/rule-tests/stellar_test.yml
(fires immediately AND is still firing 24h later — the assertion the 1h
window fails), verify_archive_textfile_test.go (seeding, accumulation,
tier isolation, atomic rename) and verify_archive_unit_wiring_test.go
(the deployed units must wire an export path — the Go↔systemd seam
lint-metric-refs.sh cannot see). Runbook, alerts-catalog and metrics
reference corrected; the never-existent producer
scripts/ops/archive-cross-check.sh is flagged as design-intent in
multi-region-topology.md. Requires an ansible apply
(`--tags ops-jobs`) on r1 — a binary-only deploy ships this dead.
Adversarial verification then found the apply procedure delivered only
HALF the fix and its confirm step read green anyway, so three further
corrections land with it: (1) the tier-b install/remove blocks in
14-stellarindex-services.yml were the only verify-archive tasks
without tags: [ops-jobs] (they sit in their own
verify_archive_tier_b_enabled conditional, added after the tag was
introduced), so the documented apply rendered tier-a's unit and
silently skipped tier-b's — leaving the CHECKPOINT tier, the
cross-archive anchor the page's own summary describes, permanently
unwired; they are tagged now and pinned by
TestVerifyArchiveUnits_ReachableUnderOpsJobsTag. (2) The runbook's
confirm is fail-CLOSED on a half-apply: it loops over both tiers and
exits non-zero naming the missing one, instead of showing tier="chain"
at 0 and deferring the other with "once tier-b has run". (3) The two
divergence checks that run OUTSIDE the per-chunk walk — cross-chunk
boundaries (stitchChunks, ~11 per 12-worker run) and the cross-run
resume seam (checkResumeFromHash) — returned their errors without
incrementing the counter, so a break landing on a chunk boundary still
paged nobody; both now increment under the same reason taxonomy
(TestStitchChunks_BoundaryBreakIsPageable,
TestCheckResumeFromHash_MismatchIsPageable, which also pins that a
malformed -resume-from-hash — operator input, not divergence — must
NOT move a severity-page counter). The runbook gained a Known blind
spots section for what is still uncovered: Tier D / Tier E emit no
metric at all, and the first run on a host with no .prom file yet
publishes a series that appears at its final value, so a break found by
that very first run reads increase() == 0 until the next run — which
is why the apply procedure now primes both units by hand. - The 7d chart column on `/assets` is back for the assets that matter
— and a withheld price no longer gets published as a picture of
itself. Rows 1–11 of the directory (XLM, USDC, PYUSD, EURC, AQUA,
yXLM, SHX, VELO, BLND, PHO, yUSDC) rendered
— in the 7D CHART column
while the unverified long tail below them charted fine. Those eleven
are exactly the catalogue-projected rows, whose wire asset_id is the
catalogue SLUG (projectCatalogueRow sets AssetID = vc.Slug), so the
listing asked GetAssetsPriceHistory7dBatch for a series under xlm /
aqua — ids that can never match a prices_1m row. The batch query
answered with its want × days skeleton: seven buckets, every price
null, indistinguishable on the wire from "this asset has never traded",
which is why it shipped unnoticed. The series now keys on the row's
Stellar twin asset_id — the SAME id its price_usd and
change_7d_pct already come from (fillCatalogueStatsForPage,
fillGlobalPriceFromOnChain) — so the chart and the price can no
longer disagree about whether data exists. ?include=sparkline7d is
also honoured on the default /v1/assets listing, where it had been a
silent no-op (the parameter was wired only into the catalogue/classic
phases, so the issue's own repro returned a byte-identical response
with and without it). Two honesty rules go with it, in both directions:
a row with no published price gets no chart and is not even looked up —
the scam-issuer suppression and the thin-market substance gate both
leave price_usd nil before the attach runs — and on the asset DETAIL
payload price_history_24h / price_history_7d are dropped whenever
the headline price is withheld (measured on r1 2026-08-29: the flagged
JFKBANK2 and RIO details served price_usd: null beside 24 hourly and
7 daily *priced* points, and their listing rows drew a full sparkline
next to a — price cell; the last bucket of a price series IS the
number being withheld). New counter
stellarindex_api_sparkline7d_rows_total{result="served"|"empty"} plus
a once-per-request warn when every priced row on a page comes back
empty: a map hit from the batch reader is not evidence of data, and
nothing anywhere reported the dead column. (#355)
- r1's ZFS `data` pool is raidz1 everywhere, and a lint keeps it that
way. The pool is SINGLE parity — live-verified 2026-07-17 and
corroborated by arithmetic that needs no host access (the ~16.8 TB
footprint measured that day cannot fit the ~13.85 TB two parity drives
would leave on these four devices) — but the 2026-07-17 correction only
ever reached the two rule trees and two runbooks.
r1-deployment-state,
self-hosting, storage-considerations, multi-region-topology,
multi-region-cutover, r3-deployment-state, lcm-cache-tiering,
ADR-0016, ADR-0027 and the ansible per-region comment all still said
raidz2, i.e. promised an operator a second drive of failure margin that
does not exist and sized capacity plans off a usable figure ~4.5 TB too
low. Sharpest of them: configs/ansible/inventory/r1.example.yml said
zfs_data_pool_type: "raidz2", so a rebuild from the codified inventory
would have laid down a pool too small for r1's own data. That value is
now raidz1 and is the machine-readable authority scripts/ci/lint-docs.sh
§18 lints every r1-scoped file against (paragraph-level: naming another
raidz level is allowed only alongside the live one, so dated history
survives and bare contradictions do not). The role DEFAULT stays raidz2
— deliberately, it is the right shape for a *fresh* archival node — and
now says so. The TODO(ash) in zfs-degraded.md is closed with the
evidence rather than another ssh request. Dated decision records
(ADR-0016/0027, the superseded first-node runbook, the 2026-07-16 audit
assessment that inferred raidz2 from docs while stating it had no live
access) keep their text and carry inline corrections. (#289) - Account transaction history no longer truncates itself: the keyset
merge dedupes the two arms BEFORE taking its LIMIT.
/v1/accounts/{g_strkey}/transactions resolves its page keys from a
UNION ALL of the sourced (stellar.ops_by_source) and participant
(stellar.operation_participants) arms, and a tx the account SOURCED
that ALSO carries it as a non-source participant of one of its
operations is emitted by BOTH arms. The merge took its LIMIT ? over
those still-duplicated rows and only the hydration pass deduped, so
every overlapping tx cost a page slot: the page came back SHORT while
older history remained, and the handler emits next_cursor only on a
FULL page (the documented "absent on the last page" contract), so a
client's history walk stopped there with older transactions
unreached. Measured on the live-ClickHouse fixture (600 txs, page
size 7): the pre-fix query served 100 non-final short pages out of
101 — a walker stopped on page 1 having seen 6 of 600 txs — the fixed
query serves 86 full pages and 0 short ones. LIMIT 1 BY ledger_seq,
tx_index now runs at the merge too, making the keyset exactly
min(limit, distinct keys older than the cursor); rows and their
order are unchanged (the integration walk keeps the pre-fix SQL as a
differential oracle over the whole history, and now requires it to
still produce a short page so the fullness assertion cannot go
vacuous). The sibling operations listing needs no such dedupe — its
arms are disjoint at op granularity because participants exclude
their op's own source — and a key cannot hydrate to nothing either,
since Sink.Flush writes transactions before operations and
participants. (#290)
- The `ops_batch` ClickHouse identity can no longer reach a live
daemon on a `deploy/systemd` self-host. The three reference units
(
stellarindex-{indexer,aggregator,api}.service) source
/etc/default/stellarindex-ops — the indexer for its MinIO creds, the
API for its SEP-10 seed — and
docs/operations/clickhouse-ops-batch-profile.md prescribes writing
STELLARINDEX_CLICKHOUSE_OPS_USER/_PASSWORD into exactly that file.
Since #243 those two vars set the identity of every ClickHouse
connection internal/storage/clickhouse opens, so a self-hoster
following the doc demoted the LIVE ledger sink
(NewLiveSink → Open) and the aggregator's supply readers
(NewExplorerReader) to the lowest-priority batch tier — the precise
inverse of the 2026-08-28 r1 incident the profile exists to prevent.
The only guard was a unit-file comment (config-assertions.sh's third
leg checks /etc/default/stellarindex, which does not exist on such a
host). Each live-daemon unit now carries
UnsetEnvironment=STELLARINDEX_CLICKHOUSE_OPS_USER
STELLARINDEX_CLICKHOUSE_OPS_PASSWORD, which systemd applies after
every Environment=/EnvironmentFile= (systemd ≥ 235; the Ubuntu
22.04/24.04 targets ship 249/255), so the guarantee holds whatever the
operator puts in the file. The batch one-shots that share the file
(verify-archive-tier-*, ch-schema-*, restore-drill) deliberately
do NOT strip it. TestOpsBatchIdentityNeverReachesLiveDaemons resolves
the environment each unit in deploy/systemd/ and the archival-node
role would hand its process, feeds it to opsAuthFrom and pins both
halves — CH default for the live daemons, ops_batch for the batch
units — so neither direction can drift. Ansible-managed hosts were
never affected (their daemons read /etc/default/stellarindex) and
nothing about r1's rendered units changes. (#292) - `stellarindex_ingestion_duplicate_flood` can fire in its own target
scenario. The rule joined
rate(...{outcome="duplicate"}[10m]) > 0.5
with and on (source) rate(...{outcome="new"}[10m]) == 0, and an
and join needs the right-hand series to EXIST.
stellarindex_trade_insert_outcome_total is call-site-seeded
(WithLabelValues on the trade-insert path) and its source label is
config-dependent, so internal/obs does not pre-seed it — a source
whose every insert since process start hit the conflict path never
creates the outcome="new" child, the join matched nothing, and the
alert stayed silent in exactly the post-restart cursor-replay flood it
exists for. Now unless on (source) rate(...{outcome="new"}[10m]) > 0,
which reads an absent child and a zero rate identically (the
absent-series idiom already used by the insert_stale sibling). Both
rule trees. The promtool case that pinned the gap as a KNOWN GAP now
asserts the alert fires, with a companion guard proving a
below-threshold duplicate rate with the same absent child stays silent
(red on the pre-fix rule: got:[]). (#302) - `/v1/livez/lake` single-flighted, and its 503 no longer publishes the
ClickHouse endpoint (#310, audit 2026-08-29). #266 gave the ADR-0050
lake-route LB probe readyz's infra exemptions — no auth, no anonymous
rate limit — but readyz's safety under those exemptions comes from its
single-flight cache, which this route never had: every anonymous
request ran a fresh
LakeTipLedger query against ClickHouse under a 5s
timeout, so unmetered concurrent probes amplified straight onto the
lake, worst exactly when the lake was already struggling. Concurrent
callers now share ONE ping round per second (livezLakeTTL, the same
budget as readyz; as_of reports when the lake was actually pinged),
and the round runs on a detached context so one prober's disconnect
can't cancel it for everyone. The 503 body's data.detail is now a
fixed operator hint instead of err.Error() — which on the real
checker is a dial error naming the ClickHouse host:port, served
unauthenticated during an outage; the underlying error goes to the
server log (once per round). OpenAPI 503 contract + the generated
Postman/TS mirrors updated to match. Tests:
TestLivezLake_SingleFlightSharesOnePingPerRound (25 probes → 1 ping;
pre-fix 25), TestLivezLake_UnreadyBodyDoesNotEchoPingError (body
scrubbed, log still carries it), TestLivezLake_RoundRefreshesAfterTTL
(a recovered lake is not pinned 503), TestLivezLake_AbsentLakeFailsClosed.
- A future-dated backup stamp no longer renders a green "fresh" row on
the status page.
freshnessVerdict
(internal/api/v1/diagnostics_backups.go) clamped a negative age to
age_seconds: 0 for display and then judged the SLO on the *clamped*
value — *age > slo is false for any negative — so a forward-skewed
host clock or a corrupt future-dated pgBackRest label painted an
arbitrarily stale backup "ok" with a 0-second age, and the panel's
roll-up went fully green (its own comment already said "don't reward
it"). A stamp past backupClockSkewTolerance (1 min — these ages
cross clocks, so ordinary NTP divergence on a genuinely fresh item
still floors to 0) is now "unknown" carrying its RAW negative age
for diagnosis, which also drags freshness.overall and flags.stale
off all-clear. The Backups panel names that state ("stamp from the
future") instead of an ambiguous grey "no data", and its client-side
repositories caption stopped clamping the same future stamp up into
"0s ago" (Math.max(0, …) — the identical bug, one layer up).
Regression tests: TestBuildBackupsSnapshot_FutureDatedOffsite (an
8 d 14 h future-dated repo2 label read ok / 0 s / overall ok
before the fix) + BackupsPanel.test.tsx. (#311) - `usd-volume-restamp`'s lifted decompression cap can no longer ride the
pooled connection out of the call.
RestampExactTierUSDVolume raised
timescaledb.max_tuples_decompressed_per_dml_transaction = 0 with a
session-level SET on a borrowed *sql.Conn, and its comment claimed
that was "session-scoped … must not leak into the pool's serving
connections". Mechanically it was the opposite: Conn.Close returns the
connection TO the pool and pgx v5 stdlib's default ResetSession is a
no-op (it pings and discards a conn left mid-transaction; it issues no
DISCARD ALL/RESET ALL), so the lifted cap persisted on that pooled
connection for the process lifetime and any later DML landing on it
would have run uncapped — harmless only because stellarindex-ops is a
one-shot whose pool never serves the API. The restamp now runs its
window in ONE explicit transaction with SET LOCAL, the same tx-scoped
GUC discipline as FindPerSourceLedgerGaps /
SEP41SupplyEventKindResum: Postgres unwinds it at COMMIT/ROLLBACK, so
it cannot escape even on the error path. Behaviour of the write itself
is unchanged (same predicate, same identity, same INV-3 generation
guard). Unit test
(TestRestampExactTierUSDVolume_DecompressionCapNeverEscapesTheTransaction,
a GUC-scoping driver fake that also models pgx's no-op session reset)
+ the DB-backed integration test now asserts the pooled conn's cap is
untouched after a restamp and that TimescaleDB honours the SET LOCAL
form inside the transaction. (#312) - The required `lint` check no longer downloads a JSON schema from
golangci-lint.run on every PR.
golangci/golangci-lint-action
defaults verify: true, which runs golangci-lint config verify —
and that command fetches
https://golangci-lint.run/jsonschema/golangci.v2.11.jsonschema.json
before it can validate anything. On 2026-08-28 the fetch died with
read: connection reset by peer and took a REQUIRED check red on a
diff containing no Go (PR #275); a rerun passed. Reproduced locally
against v2.11.4 with the network blocked (HTTPS_PROXY to a dead
port): compile schema: failing loading "…golangci.v2.11.jsonschema
.json" … connection refused, exit 3. The schema check is NOT
dropped — golangci-lint's own loader silently ignores unknown keys
(a top-level runn: block is accepted by golangci-lint run and
rejected only by config verify), so losing it would mean a
misspelt setting is a lint rule that quietly stops applying. Instead
the schema is vendored (scripts/ci/golangci.v2.11.jsonschema.json,
byte-identical to the site's copy) and validated offline by a new
scripts/ci/lint-golangci-config gate wired into the lint job,
make lint-golangci-config and verify.sh. The gate also enforces
its own preconditions: every golangci-lint-action step must set
verify: false (so the fetch cannot silently return), ci.yml and
the Makefile must pin the same release, and that release must have a
vendored schema (a bump without a re-vendor fails rather than
validating against a stale copy). Fails loudly if the action step
disappears, so it can never pass vacuously. (#317) - The MEV liquidation-cascade path stops treating unmapped `raw:`
oracle rows as evidence — a squash merge had silently reverted the
guard.
af5a9d1d (#305, a pgBackRest ansible change whose base
predated 2ce680f3) landed a tree that removed PR #248's oracle
capture-totality consumer guards and DELETED their tests. From that
merge until now, OracleUpdatesForMEVScan no longer carried
AND asset NOT LIKE 'raw:%' and buildCascadeCandidate no longer
called oracleRefIsMapped, so the one oracle_updates consumer with
NO asset keying — for the cascade correlator, any oracle row inside a
fill's ledger bracket is evidence — was again fed the
orientation-unknown raw:<symbol> rows the totality design records
verbatim. Both guards are restored verbatim, together with the two
deleted regression tests (mev_shape_test.go, cascade_raw_test.go)
and a behavioural assertion on the statement the store actually issues
(TestOracleUpdatesForMEVScan_ExcludesRawRowsFromTheIssuedSQL), which
survives a refactor away from the query const. Red-proven against
origin/main's own files at 0f13aa14: twelve raw:NOTACOIN rows
and no mapped row at all minted a complete liquidation_cascade
event naming four real accounts on the public /v1/mev feed.
Still reverted by the same merge and deliberately NOT restored
here — each needs its own change, and the v0.48.0 entry describing
them is ahead of the code until then: internal/divergence/oracle.go's
unmapped-row refusal (+ its oracle_raw_test.go); the
-- totality: includes unmapped markers and the "Unmapped feeds" KPI
in internal/storage/timescale/{oracle,bespoke_oracle,diagnostics,
protocol_stats}.go (+ the bespoke_oracle_shape_test.go assertion);
the repo guard TestOracleUpdatesQueriesDeclareRawRowPolicy
(oracle_updates_query_guard_test.go); and
test/integration/oracle_raw_consumers_test.go. - `ListMEVEvents`' doc comment claimed a cap it does not apply. It
said "limit is capped at 500"; an out-of-range limit actually falls
back to the 50-row default, which is the package's convention
(
ListIssuers, ListFreezeEvents, ListDivergenceLatest) and is
unreachable from /v1/mev anyway (parseExplorerLimit 400s an
out-of-range ?limit=). Comment corrected to the real contract and
pinned by TestListMEVEvents_LimitNormalisation; behaviour unchanged. - Agent-orientation docs re-swept against HEAD; the two claims a
machine can re-derive are now CI-enforced (`lint-docs.sh` §18,
issue #326). CLAUDE.md's repo map still located the shipped status
page at
web/status/ — it lives in the explorer at
web/explorer/src/app/status/ (stellarindex.io/status) and
web/status/ is a redirect-only Cloudflare Pages stub 301-ing to it
— and AGENTS.md still carried the make dev ("docker-compose up the
full stack"; dev.yaml has only Timescale/Redis/MinIO) and
make docs-all ("+ obs/*.go metric Name: fields"; docs-metrics is
an explicit no-op) descriptions that #259 had already corrected in
CLAUDE.md. New §18a asserts AGENTS.md's quick-start block is a
VERBATIM subset of CLAUDE.md's — duplicated prose is what drifts, so
shorten by dropping a line, never by rewording one — and §18b asserts
that an orientation doc naming web/status also names where the page
actually lives, self-disarming if the stub redirect ever goes away.
Also documented, from the code: the oracle capture-totality bullet
(reflector/redstone/band record an unmapped symbol VERBATIM as a
record-layer raw:<symbol> row instead of dropping the slot), and a
note that an explicit issues-only / one-batch-PR agreement overrides
CLAUDE.md's default long-session commit→merge→next cadence. - One unrepresentable RedStone `feed_id` no longer discards the entire
`write_prices` batch. Since the oracle capture-totality change an
unregistered feed_id becomes a
raw:<feed_id> row, and the raw
validator's refusal (empty / > 64 bytes / a byte outside printable
ASCII 0x21-0x7E) escalated to ErrMalformedPayload for the WHOLE
event. The "impossible for an ScSymbol" justification was copied
from the Reflector/Band paths, but RedStone feed_ids arrive as
ScString — arbitrary bytes, unbounded length — and write_prices
batches every updated feed into one event, so a single bad feed_id
took all ~19 feeds dark until a code change: strictly worse than the
pre-totality per-entry skip and the inverse of the totality goal.
The decoder now drops that ONE slot (surviving feeds keep their
original op_index) and records it on the new
stellarindex_source_unrepresentable_symbols_total{source} counter
plus a WARN naming the slot — deliberately NOT the unknown-symbols
counter, whose contract is "recorded as raw:" and which would send
operators hunting for rows that do not exist. New alert
stellarindex_ingestion_oracle_unrepresentable_symbols (both rule
trees, promtool scenarios) and a runbook section. (#291) - RedStone SolvBTC NAV feeds are quoted in their reserve asset, not
`fiat:USD` (D8).
SolvBTC_FUNDAMENTAL and
SolvBTC.BBN_FUNDAMENTAL publish net asset value as a RATIO
against the asset each token is a claim on, but feedRegistry
registered both against fiat:USD — so
/v1/oracle/streams?include_unmapped=true served, with
mapped=true, crypto:SolvBTC.BBN_FUNDAMENTAL quote=fiat:USD
price=1.00000000 and crypto:SolvBTC_FUNDAMENTAL quote=fiat:USD
price=1.00295305 alongside their own _USD siblings at
78313.02974310 (live r1, 2026-08-29): a BTC-backed token
published as worth $1.00. Never reached a published price
(RedStone is ClassOracle / IncludeInVWAP=false), but it was
public. Quotes are now crypto:BTC for SolvBTC_FUNDAMENTAL
(NAV_USD ÷ BTC_USD = 1.00295) and crypto:SolvBTC for
SolvBTC.BBN_FUNDAMENTAL (exactly 1.00000000 on three captures
— lake ledger 60104689, 2026-07-27, 2026-08-29 — with a NAV_USD
equal to SolvBTC's, i.e. 1:1 with SolvBTC, not BTC). Base codes,
prices, decimals and the *_FUNDAMENTAL/USD feeds are unchanged;
only the mislabelled denominator moved. The four _FUNDAMENTAL
feeds whose reserve really is dollars (BENJI, iBENJI, USST,
savUSD) keep fiat:USD. Class guard:
TestFeedRegistry_NAVFeedsQuoteTheirReserveAsset (a bare
_FUNDAMENTAL feed may carry a fiat quote only via an evidenced
attestation entry) +
TestFeedRegistry_SuffixedFeedNeverSharesQuoteWithItsBareSibling
(an X/<FIAT> feed and its bare X sibling can never share a
quote) + TestDecode_SolvBTCFamily_NAVRatiosQuotedInReserveAsset
(the live 2026-08-29 values end-to-end). ADR-0028 §2/§3 and
ADR-0014's amendment note are amended with the evidence.
Operator note: oracle_updates rows written before this
change carry the old fiat:USD label for these two feeds; the
observed values are correct and unchanged, only the label was
wrong. Any corrective relabel is a separate, operator-run data
change. - An operator-initiated projector replay is no longer a multi-hour lag
ticket that also masks a real lag (#325).
stellarindex_projector_lag_high fired on r1 at 10:24Z on 2026-08-29
for the whole ~4h of the reflector-fx replay that rewound the cursor
2,574,496 ledgers ON PURPOSE (the VES/XAU served-row deficit, Δ=97,826)
— it told the operator nothing they had not just done, and any genuine
lag on that source was indistinguishable from it for the duration. The
replay tool already records the rewind (projection_dirty_windows,
migration 0125), so the projector now publishes
stellarindex_projector_replay_window_active{source} (1 while the
cursor is inside a recorded window and still below its pre-rewind
position, refreshed every 30s from ONE query for all sources) and the
lag rule carries unless … == 1. Not a silence: the new
stellarindex_projector_replay_stalled tickets when a replay STOPS
advancing (lag still over the same 256-ledger bound lag_high uses and
not falling for 15 min inside the window, for 5 min), the excuse
expires with the catch-up rather than with the day-long dirty-window
row, a dirty-window read error publishes 0 for every source (fail open
toward alerting), and an indexer that publishes no flag leaves the lag
rule exactly as it was. The flag is gated on the recorded window's
PROVENANCE (timescale.ProjectionDirtyWindow.IsProjectorReplay, one
shared definition of the reason format for both writers): the same
table is also written by projected-rebuild -write, whose range is NOT
bounded below the live cursor (-to defaults to it, the one-writer
guard admits liveLastLedger >= to, and -allow-live-overlap bypasses
the guard — used on r1 2026-07-27), so a cursor-only bound would have
suppressed the lag ticket for a source HELD at such a window with no
operator rewind on record. The cursor bound is exclusive at the top, so
a projector wedged exactly at its pre-rewind ledger stays alertable. Go
tests (internal/projector/replay_window_test.go, 11 cases incl. both
rebuild-window probe shapes, both cursor boundaries and the fail-open;
internal/storage/timescale/projection_dirty_window_reason_test.go
pins the reason format byte-for-byte so rows already in the table
classify correctly) + promtool cases (normal lag fires / climbing
replay silent / stalled replay fires / caught-up source inside a window
raises nothing / flag absent still fires). - Gap detector pre-registers `runs_total` at 0 so a restart cannot read
as a dead detector.
stellarindex_ingest_gap_detector_runs_total is a
CounterVec that only materialises a series on first Inc(), and since
the schedule is seeded from the persisted scan cursor (v0.49.0) a
restart legitimately runs NO scan for hours. The whole family was
absent for that window, so stellarindex_ingest_gap_detector_silent's
absent_over_time(runs_total[15m]) clause fired at 09:55Z on r1 on
2026-08-29, 26 min after the v0.49.0 deploy restarted the aggregator —
sum by (outcome)(runs_total) returned no series at all. RunGapDetector
now seeds {outcome="ok"} and {outcome="error"} for every configured
target at start (same F-0033 contract as obs.seedBoundedLabelSeries),
so the absent clause is reserved for the process-dead case. Unit test
(TestGapDetectorPreregistersRunsTotalSeries) + promtool scenario
"restart, series at 0 for 30m, fresh stamp → silent". - A galexie restart is decided only by what the running process has
loaded, is visible in `--check --diff`, and needs an explicit ack.
On 2026-08-29 06:04Z an ansible apply (
--tags users,minio,galexie)
restarted a healthy r1 galexie — a ~9-minute mainnet captive-core cold
catchup — for a change to the galexie-append.sh wrapper, which
systemd exec's once per service start and the running process never
reads; the preceding --check --diff had shown no handler because the
#267 effective-change gate compared checksums before/after a real
write, and check mode writes nothing. tasks/galexie-effective-
checksum.yml now compares each restart-relevant input on disk with
the controller-rendered would-be file (comments/blank lines stripped),
so the verdict — and RUNNING HANDLER [Restart galexie] — appears in
check mode too; the inputs are only captive-core-galexie.cfg,
galexie.toml, /etc/default/galexie and the unit (the wrapper,
the archive-fill/tip-lag/contiguity scripts + timers and the SDF apt
key can never notify the restart); and when galexie is active a real
apply that would restart it FAILS before writing anything unless
-e galexie_restart_ack=true (default false) — the same ack gates a
galexie_version binary rebuild. /etc/default/galexie moved from
inline content: to templates/galexie.env.j2 (byte-identical
output) so the gate renders the same file the task writes.
scripts/ci/ansible-galexie-restart-test.sh pins the input list, the
ack default, the fail-closed refusal, the ack path and the check-mode
preview. Runbook: docs/operations/runbooks/galexie-catchup-refused.md
§"Applying galexie config with ansible".
- Nightly pgBackRest wrapper never backed up repo2. pgBackRest's
backup
command is single-repo: with no --repo it writes only the highest-priority
repo (repo1); only archive-push and expire fan out (User Guide, "Multiple
Repositories"). pgbackrest-backup.sh ran pgbackrest --stanza --type backup
with no --repo, so once repo2 (S3) went live on r1 (2026-08-29) it would have
received WAL forever but never a full/diff — the off-site copy would have aged
out at its 7-day retention. The wrapper now discovers every repoN-* key in
/etc/pgbackrest/pgbackrest.conf and runs one --repo=N backup per repo
(repo1 first, repo2 next; a failure does not skip the next repo; exit is the
first non-zero rc, so pgbackrest-backup.service still fails loudly), with
per-repo node_exporter textfile metrics
stellarindex_pgbackrest_backup_{last_success_unix,last_rc,duration_seconds}{repo}
(last_success_unix carried forward across a failed run). Single-repo hosts
keep the byte-identical legacy command. scripts/ci/pgbackrest-backup-test.sh
pins all of it against a stubbed pgbackrest. - `deploy.yml` migrations sync took > 16 minutes on r1 (v0.49.0, run
33244745680) — one archive transfer again, with delete semantics.
#268 replaced the migrations-dir
ansible.posix.synchronize (one
rsync, delete: true, but blind to the test-net ProxyJump) with an
ansible.builtin.copy whose src is a DIRECTORY: connection-agnostic,
but one SFTP round-trip + remote checksum per file with no
ControlPersist across module invocations on the GH runner — 291
already-identical files ran past 16 minutes where the whole deploy used
to take ~7, and stale files on the host were silently kept.
tasks/sync-migrations.yml now builds ONE deterministic tar.gz on the
controller (sorted, mtime 0, uid/gid 0, modes 0644/0755 — so
unarchive's tar --diff reports changed=false on an identical
re-run), ships it with ansible.builtin.unarchive (rides the same
connection as every module, so the jump still works), and prunes
host files absent from the controller-computed manifest — fail closed
(empty manifest or non-nested dest aborts; only paths find
enumerated inside the dest are ever removed). deploy-sync-test.sh
(ci.yml ansible-check, verify.sh) pins exactly one transfer task, no
directory-src copy / per-file loop, and runs the task file for real:
extras pruned, idempotent re-run, nothing outside dest touched, empty
source fails closed — 3 of those red against the #268 task. - `fiat:VES` and `rwa:XAU` — the two reflector-fx slots that paged
`stellarindex_ingestion_oracle_unknown_symbols` on r1 v0.48.0
(2026-08-29, `raw:VES` / `raw:XAU`, 7 rows each in 2 h). The cause
was the allow-lists, not the decoder: VES (Venezuelan bolívar
soberano, ISO-4217) joins the ADR-0010 fiat list and XAU (spot gold,
troy oz) joins the ADR-0028
rwa: list — a commodity, deliberately
not fiat and distinct from the tokenized XAUm. Via the shared
canonical.MapOracleSymbol precedence both now decode as mapped
rows at the same positional op_index (DAT-03), so the 0109
generation-guarded upsert rewrites the existing raw: rows in
place on replay; the counter no longer increments for them
(TestRealDecoder_fxVESAndXAUMappedNotRaw; the real 2026-04-23 FX
fixtures now decode with zero raw rows). The alert stays red for up
to 25 h after deploy (its increase[25h] window — runbook). The
reflector-fx replay from the first raw: ledger is declared in the
commit's Replay-Plan: trailer; pre-#247 history (slots dropped,
not recorded) is covered by #247's full re-derive (PR-7 of the
totality design), which must run on a binary carrying this change.
- Explorer test-net builds could silently serve MAINNET data.
web/explorer/next.config.mjs inlined NEXT_PUBLIC_API_BASE_URL ??
'https://api.stellarindex.io' through its env block, so the
per-network fallback added in #212 (API_BASE_URL ??
CURRENT_NETWORK.apiBaseUrl in src/api/client.ts) was unreachable:
a testnet/futurenet Pages project with NEXT_PUBLIC_NETWORK set but
the API var forgotten baked in the mainnet origin. The key is dropped
from env (Next inlines NEXT_PUBLIC_* from the build environment
on its own), useMe now shares API_BASE_URL instead of its own
mainnet-literal fallback, and the JSON-LD contentUrl on asset /
market pages derives from CURRENT_NETWORK.apiBaseUrl. The
mainnet-hardcode guard now also scans next.config.mjs and strips
// comments before /* */ — a /dashboard/* in a line comment had
opened a phantom block comment that hid three literals from it.
New src/lib/next-config-env.test.ts pins the env contract
(audit web-status-5). - Native XLM supply is now network-aware.
internal/supply derived
total_supply / max_supply from the frozen pubnet constant
(50,001,806,812 XLM) regardless of the configured network, so
api.testnet.stellarindex.io/v1/assets/native served the mainnet
figure against a testnet ledger whose total_coins is 100 B (measured
2026-08-28). The aggregator and stellarindex-ops supply snapshot
now build the computer via supply.NewXLMComputerForNetwork(cfg.Stellar.
Passphrase(), …): testnet and futurenet get the 100 B genesis total
(== their ledger total_coins), pubnet output is byte-identical, and an
unrecognised passphrase fails at startup (supply.ErrUnknownNetwork)
instead of silently falling back to the pubnet number. supply_basis
is unchanged (xlm_total_only with no reserve accounts configured,
which is the honest testnet state). Existing testnet/futurenet
asset_supply_history rows written with the 50.0 B total are
superseded by the next refresher cycle after deploy. - Explorer: ledger page captions `total_coins` (2019-burn basis).
/ledgers/{seq} printed the header's total_coins (~105.4B XLM on
mainnet) bare, while /assets/native serves the market's 50.0B total
supply — the same unlabeled 2.11× divergence the network page fixed
(#241). The caption ("ledger header · includes the 2019 burn" on
mainnet, "ledger header" on the test nets, whose genesis has no burn)
now comes from a shared lib/xlm-supply.ts helper so every surface
printing total_coins says the same thing; #241 should adopt it. - pgBackRest repo2 retention was hardcoded to 4 fulls in
pgbackrest.conf.j2, ignoring
pgbackrest_repo2_retention_full/diff (lean defaults 1 / 7 d ≈ $12–17/month); the template now
renders the variables plus repo2-retention-archive-type=diff. Caught by a masked diff of the
rendered file against r1's live config before the first off-site apply (2026-08-29). - `ops_batch` / `api_serving` ClickHouse drop-ins no longer restart
`clickhouse-server`. Both tasks in
archival-node/tasks/20-clickhouse-serving-profile.yml carried
notify: Restart clickhouse-server for a users.d drop-in that
ClickHouse hot-reloads — on r1 the first ops_batch enable
(--tags clickhouse-ops-batch-profile,minio,heavy-job-wrapper)
would have bounced the 9.3 TB lake (minutes of explorer/lake
downtime + cold caches) for nothing; caught in the --check --diff
(2026-08-29). The notify is replaced by SYSTEM RELOAD CONFIG plus a
retried assert that system.users / system.settings_profiles each
hold exactly one of the profile's entities (skipped under --check,
21-clickhouse-drop-guard.yml's pattern), so a rejected drop-in
fails the apply instead of silently keeping the old users config.
The only CH change in the role that still restarts is
08-clickhouse.yml's config.d/si-override.xml (tcp_port /
listen_host), which genuinely needs it.
Added
- Explorer /oracles opts into oracle capture-totality (PR-5 of 7).
The page now requests
/v1/oracle/streams?include_unmapped=true and
renders the raw:<symbol> rows — oracle-published symbols that map
to no canonical asset — in a separate "Unmapped feeds" section under
the raw on-wire symbol (monospace, unlinked), never mixed into the
mapped price-stream table or its per-oracle counts. A raw: id now
has a first-class rendering everywhere an asset is shown
(shortAssetText → the symbol, AssetLabel → monospace symbol,
assetSlug → no link; previously it would have linked to a
static-export 404 under /assets/raw…). The oracle source bespoke
page needs no explorer change: its counts/tables are text-only and
already totality-inclusive server-side with the "Unmapped feeds" KPI. - CI ansible task lint (`scripts/ci/lint-ansible-tasks.sh`). Two
structural guards over
configs/ansible/**, wired into import-checks +
verify.sh with a fixture self-test: *pipefail-needs-bash* (a
ansible.builtin.shell body that sets pipefail must declare
executable: /bin/bash — dash rejects it, the third recurrence of the
class) and *secret-on-argv* (a vault value interpolated into a
command body / mc positional secret in a shipped script; no_log
hides it from Ansible output only). Grandfathered violations live in
the shrink-only lint-ansible-tasks.baseline. - CI galexie restart-wiring test
(`scripts/ci/ansible-galexie-restart-test.sh`, ansible-check job).
Pins that the five galexie-input render tasks no longer notify
Restart galexie directly, that the bootstrap binary install restarts
every daemon it replaces, and exercises
galexie-effective-checksum.yml for real (local ansible run, stub
handler): comment-only edit → quiet; code/shebang edit or new file →
restart. - `stellarindex-ops usd-volume-restamp` (W5.3). The corrective WRITE
half of
verify-usd-volume: for every exact-tier (quote- or base-leg
USD-pegged) group in a bounded -from/-to day window it rewrites each
row whose stored usd_volume differs from pegged_leg / 10^decimals
to exactly the value the insert path writes, stamped with the run's
derive_generation (INV-3 guard: a live gen-0 replay can never claw a
correction back). Tier + scale come from the same
ClassifyUSDVolumeTier + peg config as the writer and the verifier —
no SQL reimplementation of the waterfall. Dry-run by default, -write
to apply, idempotent (correct rows are untouched, value and
generation), -sliced UPDATEs on a dedicated session with the
Timescale decompression cap raised, ch-backfill-style heartbeat. Replaces
the 2026-07-30 hand SQL for the pre-2026-07-23 USDC-base SDEX class
(docs/operations/usd-volume-rederive-2026-08.md step 5). Estimated
tiers stay ch-rebuild's job. Unit tests pin the formula to
tradeUSDVolume byte-for-byte; the integration test proves the SQL
identity, the differential (a correct row is unchanged), idempotency
and the generation guard on real TimescaleDB. - Oracle capture-totality consumers (PR-3 of 7): every `oracle_updates`
reader is safe for `raw:` rows before the decoders emit them.
/v1/oracle/streams gains include_unmapped (default false — the
public row set is unchanged; the explorer's /oracles page is the
intended opt-in) and OracleReading gains a required mapped flag
(false for raw:<symbol> rows; /v1/oracle/latest?asset=raw:…
returns one by its exact key). The MEV liquidation-cascade
correlator — the one unkeyed reader, for which any oracle row in the
ledger bracket is evidence — excludes raw rows both in
OracleUpdatesForMEVScan SQL (asset NOT LIKE 'raw:%') and in
DetectLiquidationCascades; the divergence OracleReference
(and, through its cache, the confidence cross-oracle factor and the
Phase-2 freeze lens) refuses a raw row as ErrAssetUnsupported on
top of its exact-string keying. The oracle source bespoke page keeps
raw feeds in every count and table (totality) and adds an
"Unmapped feeds" KPI + note. LatestOracleStreams no longer drops a
row with an unparseable asset/quote silently (it logs the row's
identity — a raw: row parses, so a miss is a malformed legacy row).
A repo guard (TestOracleUpdatesQueriesDeclareRawRowPolicy) now
requires every FROM oracle_updates literal under internal/ to be
asset-keyed, carry asset NOT LIKE 'raw:%', or a
-- totality: includes unmapped marker, so the cascade class of
unkeyed reader cannot recur unlabelled. Each surface carries a test
red-proven with a fixture raw row; storage behaviour pinned on real
Timescale in test/integration/oracle_raw_consumers_test.go. - ClickHouse destructive-DDL size guard pinned by ansible. New
archival-node/tasks/21-clickhouse-drop-guard.yml (tag
clickhouse-drop-guard) writes
/etc/clickhouse-server/config.d/si-drop-guard.xml pinning
max_table_size_to_drop / max_partition_size_to_drop to
ClickHouse's 50 GB default (clickhouse_max_*_size_to_drop, 0
refused) and asserts the live system.server_settings value; no
restart (hot-reloaded). r1 was measured at 1 TiB for both — raised
by hand for D2's REPLACE PARTITION and never lowered — so
account_movements (582 GiB) was droppable in one statement.
Planned oversize drops now use the self-deleting
force_drop_table flag after a ZFS snapshot:
docs/operations/clickhouse-destructive-ddl.md.
d2-ordinal-reproject.sh requires D2_FORCE_DROP=yes and
d3-lecur-v2-rebuild.sh rollback-precutover requires
D3_FORCE_DROP_V2=yes; both arm the flag per guarded statement
and remove it after. - Oracle capture-totality PR-2 — decoders record unmapped symbols
as `raw:` rows. The Reflector (dex/cex/fx), RedStone and Band
decoders no longer SKIP a price slot whose symbol / feed_id is
outside the canonical allow-lists (ADR-0010/0014/0028, RedStone
feed registry); the slot is recorded verbatim as a
raw:<symbol> asset (canonical.AssetOracleRaw, PR-1) at the
SAME positional op_index the skip placeholder consumed, so no
existing row's identity moves (DAT-03; pinned by
TestDecodeUpdate_OpIndexStableAcrossAllowlistState and the
RedStone/Band mixed known/unknown tests). All-unknown events now
decode to rows instead of ErrEmptyPrices / ErrEmptyUpdates /
ErrEmptyRates (those remain only for genuinely empty / all
non-positive vectors). Raw RedStone rows are quoted from a
/<FIAT> feed_id suffix when it is allow-listed, else USD, and
are never inverted (orientation unknown). Band still skips USD
and rate-0 slots. The real 2026-04-23 mainnet fixtures show the
gain: every reflector-fx event carries VES and XAU slots that
were dropped before. Shared mapper canonical.MapOracleSymbol
(fiat → crypto → RWA → raw; lists pinned disjoint) replaces the
two decoders' inconsistent precedence.
stellarindex_source_unknown_symbols_total keeps its name and
keeps incrementing; it now means "recorded as raw" (help text +
metrics reference updated). Consumers (/v1/oracle/streams,
bespoke page, MEV cascade) are PR-3/4 — a raw row is
IsMapped()==false and must be excluded there.
Completeness-gate impact: the expected side of the oracle
completeness gate is the same decoder run over the lake, so
reflector-*/redstone/band historical ranges read INCOMPLETE (Δ =
historical unmapped slots) from the moment this binary deploys
until history is replayed; the replay is declared in the commit's
Replay-Plan: trailer (the scripts/ci/lint-replay-plan.sh
convention) and executed as PR-7 of the design.
- CI replay-plan tripwire (`scripts/ci/lint-replay-plan.sh`). On
2026-08-27 e17288bd widened
internal/canonical/asset_fiat.go 32→132
codes; live ingestion recorded 4 new currencies, nobody replayed
history, and 190,228 served rows were missing for a day (surfaced
2026-08-28 by a stale gate binary upgrade). The new gate, wired into
the import-checks job next to lint-baseline-growth.sh and mirrored
in scripts/dev/verify.sh, fails any range touching
internal/canonical/asset_{fiat,crypto,rwa}.go or
internal/sources/*/{decode*,events,feeds,pairs}.go unless a commit
carries a Replay-Plan: trailer (none — <reason> allowed; a bare
none is not). Fixture self-test in lint-replay-plan-test.sh.
Changed
- Aggregator: structurally single-venue crosses (`crypto:XLM/fiat:GBP`)
keep the signed freeze-and-auto-release posture; the USD-FX-derived hub
route is never counted as a second source. In the production graph
every triangulation target has exactly one hub route (through USD), so
corroborationCount is always 1 and the freeze's source_count
widening never executes (design:
docs/design/composite-route-corroboration-for-structurally-single-venue.md
§1–§2). A candidate change that entered the target's own direct print
into the corroboration clique was rejected in review: the widening is
read one tick behind, so a prior-tick "agreement" pinned sources=2
for the NEXT bucket and a persistent single-venue manipulation was
never frozen. TestRouterFreeze_TwoRoutesSuppressSingleSourceFreeze
now pins the production shape as a control (single agreeing hub route
+ single-venue z≈50 spike → freeze engages, hold kept on the
persisting print, prevVWAPs does not ratchet, last-known-good keeps
serving). The real second route (design §8.1, crypto:XLM/crypto:BTC
in aggregate.pairs) is an operator decision because it also exposes
a new, non-min_usd_volume-gated served pair. - CI amtool gate for the Alertmanager config (#275). The routing
tree that decides whether any alert reaches a human was the only
production config surface with no CI validation — amtool ran only
inside
configs/alertmanager/apply.sh, by hand, on the host, after
merge. apply.sh grows --check-only (render + validate, no
install), the monitoring-rules job installs a SHA-pinned amtool and
validates BOTH render branches (empty URLs → the block-stripper stub
path; set URLs → substitution), verify.sh mirrors it with
promtool-style graceful skip, and configs/alertmanager/ joins
config-apply-gate.sh SURFACES. - Canonical `raw:` asset type — the record layer of the oracle
capture-totality design (PR-1 of 7).
canonical.AssetOracleRaw
(raw:<symbol>, 1–64 printable-ASCII bytes, no allow-list) holds an
oracle-published symbol verbatim when it maps to no canonical asset.
ParseAsset dispatches the prefix ahead of the classic
<code>:<issuer> split (on main raw:BTC parsed as classic code
raw + issuer BTC and failed), Validate/Value/Scan/JSON
round-trip it, and Asset.IsMapped() is the interpretation-layer
guard. Pair.Validate refuses a raw leg (never a VWAP input),
supply.AssetKey treats it as off-chain, and every oracle_updates
reader that re-parses the asset column (LatestOracleUpdatesForAssets,
LatestOracleObservation, LatestAggregatorPricesForPair,
LatestOracleStreams) now tolerates a raw row instead of failing the
request — integration-tested against real Timescale. No decoder emits
raw rows yet (PR-2); this PR changes no served behaviour. - Alert `stellarindex_ingestion_oracle_unknown_symbols` — the
2026-08-04 cold audit found
stellarindex_source_unknown_symbols_total
had no consumer in either rule tree while r1 carried 7,794 silently
dropped Reflector slots. Fires per source on any increase over a
trailing 25 h (longer than Band's daily cadence, so it cannot flap),
ticket severity, promtool unit-tested, runbook
docs/operations/runbooks/oracle-unknown-symbols.md.
Fixed
- Deploy served-path smoke (#232):
deploy.yml now curls the public
endpoints after restart and asserts the r1 version-skew probe reads 0 —
a deploy is not done until the served artifact answers. - Ops-job unit tasks are taggable (#229):
--tags ops-jobs applies the
heavy-job wrapper/timers without the whole archival-node role. - CI: `timeout-minutes` on every previously uncapped job (23 jobs, 9
workflows, #260) and
persist-credentials: false on all 34 checkout
steps (#276).
Fixed
- `GET /v1/accounts/{g}/operations` and `/transactions` pages for a
hot account no longer cost the account's whole history. Each UNION
arm resolved its keys over
stellar.operations /
stellar.transactions with pk IN (SELECT pk FROM ops_by_source …);
ClickHouse prunes that to one granule PER KEY in the set before the
LIMIT, so a page for an account with 11,925 sourced + 26,064
participant ops read 164–238 M rows (~8 s, AccountOperations
deadline exceeded 503s on r1, 2026-08-28). The arms now page
stellar.ops_by_source / stellar.operation_participants directly
(both keyed (account, ledger_seq, tx_index, op_index), so the
cursor, watermark bound and LIMIT apply on a primary-key-prefix
range read) and the wide table is touched once, by the hydration
pass, over ≤ 3×limit keys. Rows, ordering, cursor semantics and
source/participant dedupe are unchanged (differential integration
test against the previous SQL, plus a system.query_log read_rows
bound).
- RedStone docs (
internal/sources/redstone/README.md,
docs/protocols/redstone.md) cited a metric that never existed
(redstone_unknown_symbols_total); the counter is
stellarindex_source_unknown_symbols_total{source="redstone"}. - Low-priority `ops_batch` ClickHouse identity for heavy
`stellarindex-ops` jobs (2026-08-28 r1: a runbook-prescribed
ch-rebuild -sep41 dry-run running as CH's default user starved the
aggregator's supply refresher — supply_refresh_error_dominant for
all 39 watched contracts in 3 minutes; the cgroup caps could not help
because the contention was inside clickhouse-server). Every
ops-side ClickHouse connection builder in internal/storage/clickhouse
now takes an optional username/password from the environment —
new env vars STELLARINDEX_CLICKHOUSE_OPS_USER /
STELLARINDEX_CLICKHOUSE_OPS_PASSWORD (never argv; both unset =
unchanged default-user behaviour). New ClickHouse user + settings
profile ops_batch (priority=100, os_thread_priority=5,
max_threads=2, 8 GiB, readonly=0), provisioned by
20-clickhouse-serving-profile.yml behind
clickhouse_ops_batch_profile_enabled (default false; enabling
asserts vault_clickhouse_ops_batch_password). run-heavy-job.sh
imports the pair from /etc/default/stellarindex-ops into every
wrapped job itself and prints the identity it will use (or a
WARNING ... CH 'default' user line), and config-assertions.sh
asserts the CH user and the env pair moved together. Operator
steps (per host, one PR): add the vault password + flip the flag in
the inventory, then ansible-playbook ... --tags
clickhouse-ops-batch-profile,minio,heavy-job-wrapper --check --diff,
then apply (the third tag re-renders the wrapper already on the
host). Pinned by TestOpsOpenersAuthenticateFromEnv (the identity
every opener puts on the ClickHouse wire, decoded from the native
client hello) and scripts/ci/run-heavy-job-test.sh (the shipped
wrapper, extracted from the ansible task). See
docs/operations/clickhouse-ops-batch-profile.md.
- Backup alerts could not fire on the absent-series case (audit
2026-08-28, backup-restore-1 / backup-restore-2). Both rule trees.
stellarindex_timescale_backup_none_24h / _failed are min by
(stanza)(pgbackrest_backup_since_last_completion_seconds…) > N, which
is an empty vector — never fires — when pgbackrest_exporter is up
but has no stanza to report (pgbackrest not installed, conf/repo
unreadable by the exporter user, stanza error), and the only backstop
checked up alone. New stellarindex_pgbackrest_backup_metrics_absent
(page: up == 1 unless on (instance) <backup series>) and
stellarindex_pgbackrest_backup_unit_failed (ticket) close it.
Likewise stellarindex_ch_schema_snapshot_stale / _offsite_stale
were time() - last_success > N over a stamp the script drops on any
partial run (and never writes on a first-run failure), so the
never-succeeded / every-run-failed cases were unalerted: both gain the
OBS-2 absent_over_time branch (offsite gated on the new
stellarindex_ch_schema_snapshot_offsite_configured gauge so acked
local-only hosts stay silent) plus stellarindex_ch_schema_snapshot_unit_failed.
Promtool tests proven red on origin/main:
deploy/monitoring/rule-tests/storage-backup_test.yml. Also corrects
the stale stellarindex_pgbackrest_last_success_unix name in
meta.yml / the alerts catalogue.
- Status page rendered stale / unreachable state as fresh green
(web-status-1/2/4/6, audit 2026-08-28).
/status (a) dropped the
/v1/diagnostics/ingestion envelope's flags.stale, so a failed
cursors/network-stats read (zero-valued fields) painted "Lag from tip
0s" in green and "Latest ledger 0" — and on the lean nets fed a 0 s
lag into the indexer roll-up; (b) kept "All systems operational" and
a pulsing green "Live" for as long as a tab stayed open during a total
API outage, because the headline was derived from the retained
last-known snapshot with no regard for the poll error; (c) labelled
the coverage table with backfill_coverage_as_of (the API's
per-request assembly time, "4s ago" forever) while the completeness
verdicts it shows are dated by a daily timer, and claimed the detector
runs "every 5 min" (it is 30 min); (d) blanked operator notices the
moment the notices endpoint failed — the exact window a notice
announces. Now: stale snapshots carry a "stale · server degraded"
badge, unmeasured zeros render "—" and never vote in the lean-net
roll-up; after two failed polls (the DegradedBanner threshold) the
headline is "Status unknown", the pulse goes grey and reads "last
successful poll <age>", and the unreachable card sits above the
headline; each coverage row shows its real data age (30 min axis for
the gap detector, daily axis for compute-completeness) and an aged
verdict loses the green verified tone; notices are retained through
a failed poll with a "notices feed unreachable" marker and cleared
only by a successful response. Red-proof tests in
web/explorer/src/app/status/StatusPageClient.test.tsx. - `pgbackrest.conf` template task printed the backup cipher passphrases
and repo2 S3 key pair on `--check --diff`. The task rendering
pgbackrest.conf.j2 had no diff: false, and the header comment
claimed the credentials came from env vars "never inline in this
file" — they are (and must be: pgbackrest reads them from the
postgres-owned 0640 file, including from archive_command inside the
postgres server process, which no unit-level EnvironmentFile reaches).
The documented review path (--check --diff in README, the operator
register, and the weekly ansible-drift job which tees its output
into a GitHub Actions log — where only registered secrets are masked,
not vault-decrypted values) therefore printed the ONLY key that
decrypts the offsite survival backup. The task now sets diff: false
(changed/ok verdict stays visible; hunk body suppressed) and the
header tells the truth. New gate scripts/ci/lint-ansible-secret-diff.py
(ansible-check job + verify.sh) fails any template task whose
template renders a secret-shaped var without diff: false/no_log;
the seven pre-existing sibling tasks (redis, patroni, keepalived,
minio) are grandfathered by name for burn-down. Retention is
untouched (ADR-0043). (audit-2026-08-28 backup-restore-7)
- Restore-drill capacity floor was a 200 G constant under a ~600 G
restore; a partial restore was kept on the shared pool.
scripts/ops/restore-drill.sh now sizes its free-space requirement
from the latest backup in the stanza/repo it is about to restore
(pgbackrest info database size × DRILL_SIZE_MARGIN_PCT=125 % +
DRILL_WAL_HEADROOM_GB=50 G, clamped up to MIN_FREE_GB); a backup
that cannot be sized is a precondition refusal (exit 2). A
pgbackrest restore that does not complete has its partial datadir
removed unconditionally (the pgbackrest output is the diagnostic);
post-restore failures still keep the datadir. (audit 2026-08-28
backup-restore-3; the ZFS quota on data/restore-drill is a separate
ansible change.) - Restore-drill abort paths left the previous PASS being scraped.
A failed
pgbackrest restore or a scratch instance that never reached
consistency exited before the evidence phase: no drill-log entry,
and last month's restore_drill.prom (failures 0, fresh
last_success) kept being served until the 40-day staleness ticket.
Evidence + metric emission are now functions called on every path
past the preconditions; aborts write ABORTED at <stage> and
failures N with no last_success. New ticket alert
stellarindex_restore_drill_failed (failures > 0 for 30 m) in both
rule trees + promtool rule-test + runbook. New
scripts/ops/restore-drill-run-test.sh drives the real script through
the capacity-refusal / restore-failure / recovery-failure paths with
shimmed host tools; it and restore-drill-test.sh are now wired into
CI + verify.sh. (audit 2026-08-28 backup-restore-4) - Config-apply gate: widened surfaces, host baseline, fail-closed
(`scripts/ci/config-apply-gate.sh`, audit deploy-ansible-gate-4). The
gate's SURFACES list omitted files the archival-node role renders or
copies onto the host —
roles/*/defaults/ (e.g. galexie_ledgers_per_file
→ galexie.toml.j2), handlers/, inventory/, the sibling roles,
configs/healthchecks/* and the copied scripts/ops/config-assertions.sh,
ch-schema-snapshot.sh, restore-drill.sh, scripts/dev/r1-smoke.sh —
so a defaults-only or healthchecks-only release passed as "binary deploy
is complete". It also diffed against the previous tag by ancestry only
(a skip-ahead deploy never listed the skipped tags' config) and turned
any git error into "no changes" (|| true under no set -e). Now the
whole configs/ansible/roles/ tree, inventory/, configs/healthchecks/
and the four copied scripts are surfaces; an optional 3rd argument takes
the host's live version as the baseline (ancestry remains the fallback,
and the baseline used is printed); an unresolvable version/baseline or a
failing git diff exits 1. config-apply-gate-test.sh pins one fixture
per surface plus the skip-ahead and fail-closed cases (13 red on the old
script). Wiring the host sidecar
(/var/lib/stellarindex/deployed-versions/<binary>) into deploy.yml as
that 3rd argument is a follow-up. - `deploy.yml` could never apply schema on testnet/futurenet
(deploy-ansible-deploy-2).
deploy-binary.yml synced the migrations
dir with ansible.posix.synchronize, whose rsync runs on the
controller and never sees the --ssh-common-args "-o ProxyJump=…" the
NAT-only VMs are reached through, and its fail-closed pgBackRest
freshness gate can only exit 1 on a VM that has no pgbackrest at all —
so the only passing test-net deploy was migrations_skip=true
(binary on a stale schema while /v1/healthz is 200). The sync is now
ansible.builtin.copy (connection-agnostic, same as the role's own
migrations sync), the gate is conditioned on a new
pgbackrest_backup_enabled extra-var (default true — fail closed)
that deploy.yml sets per region (false only for testnet/futurenet,
with a loud "no verified recovery point" notice), and
scripts/ci/lint-deploy-playbook.sh + a CI --syntax-check of
deploy-binary.yml pin both properties. r1's path is unchanged.
- systemd EnvironmentFiles were `.`-sourced by the deploy migrate step,
a root cron and twelve host scripts (deploy-ansible-secrets-5).
/etc/default/stellarindex, -ops and /etc/default/galexie are
rendered unquoted (what systemd wants), but every shell consumer
re-parsed them with set -a; . <file> — so a secret carrying $, ;,
&, |, quotes or whitespace works under systemd and is silently
mangled (or its tail EXECUTED as root) on the sourcing paths: migrate
fails 28P01, ARCHIVE_TO stays 0, freshness gauges go stale. Every
consumer now reads the file verbatim (the run-heavy-job.sh pattern —
one canonical load_env_file in the bash scripts, a POSIX loop in the
/bin/sh migrate task and the cron one-liner), and a preflight assert
refuses to render any of the sixteen env-file secrets that contains a
shell metacharacter (names only in the message; rotate to
openssl rand -hex 32), which also makes the documented interactive
set -a; source safe. scripts/ci/envfile-loader-test.sh pins all
consumers in lockstep and round-trips a metacharacter fixture.
- `deploy.yml` `health_grace_seconds` was unvalidated and spliced into
`ansible-playbook -e` (deploy-ansible-input-8). The input is
type: string (the comment claiming GitHub enforced number was
false), and ansible's k=v -e form splits on whitespace, so a
dispatch value like 15 backup_freshness_skip=true injected a second
extra-var and skipped the backup-freshness gate with a nominal-looking
step summary. Validate now requires ^[0-9]{1,4}$;
scripts/ci/deploy-inputs-test.sh runs the shipped Validate step
against the injection and the malformed cases.
- ansible: `04-users.yml` history-archive sweep aborted every full
archival-node apply. The 2026-08-27 task ran
set -euo pipefail under
ansible's default /bin/sh (dash on every target) → set: Illegal
option -o pipefail, rc=2, play aborted at step 04 before
postgres/galexie/services. Now runs under executable: /bin/bash
(audit 2026-08-28, deploy-ansible-shell-1). - ansible: MinIO secrets no longer on `mc` argv.
09-minio.yml's
alias task claimed env-based auth while passing the root password on
argv, and the three mc admin user add tasks + galexie-append.sh's
per-restart mc alias set put the writer secrets in
/proc/<pid>/cmdline. All five now feed the secret on stdin (mc
reads omitted keys from stdin); the persisted local alias the root
ops scripts rely on is unchanged (deploy-ansible-secrets-9). - ansible: galexie restarts only on effective config change. The
wrapper script, captive cfg,
galexie.toml, /etc/default/galexie
and the unit are each ~50% comments, and every byte change notified
Restart galexie — a ~9-minute mainnet cold catchup per fire. New
galexie-effective-checksum.yml hashes each input with comments/blank
lines stripped before and after rendering and notifies the restart
only on a mismatch; a real change (rotated key, PEER_PORT, ExecStart)
still restarts — no default-off ack gate (deploy-ansible-handlers-7). - ansible: bootstrap binary install (`manage_stellarindex_binaries`)
restarts api + aggregator too, refuses a dirty tree, writes sidecars.
It notified only the indexer, so the api unit kept the old binary in
memory; it built from whatever was in the operator's checkout with no
record. Now:
git status --porcelain must be empty (override
-e stellarindex_bootstrap_allow_dirty_tree=true for a deliberate
unreleased-branch bring-up), and each binary's
/var/lib/stellarindex/deployed-versions/ sidecar records
git describe --tags --always --dirty so the next deploy.yml labels
its rollback copy truthfully instead of untracked-<ts>
(deploy-ansible-drift-3). - Operator self-service key mint/revoke now audited (api-security-1,
audit 2026-08-28).
POST /v1/account/keys copied an operator
caller's tier verbatim into the child and recorded nothing — no
X-Reason, no key.mint row — so a compromised staff credential
could spawn further operator credentials that POST /v1/admin/keys
would have refused without a reason and logged. Tier inheritance
(staff rotation) is kept; operator-tier callers of
POST /v1/account/keys and DELETE /v1/account/keys/{keyID} now
need X-Reason (400 without) and land the same key.mint /
key.revoke audit rows as the admin routes. Customer-tier callers
are untouched. - Rotated signup keys no longer 403 forever under email verification
(api-security-2, audit 2026-08-28). With
signup_require_email_verification on (the default), a verified
/v1/signup customer who rotated via POST /v1/account/keys got a
signup-<hash> child with a zero EmailVerifiedAt, and nothing can
verify a non-signup KeyID after the fact — RequireEmailVerified
rejected the child permanently (and revoking the parent stranded the
customer). auth.CreateAPIKeyRequest gained EmailVerifiedAt; the
self-service mint copies the caller's stamp onto the child. Signup
and admin mints still leave it zero. - `/v1/livez/lake` added to the unauthenticated-infra exemption and
the anonymous rate-limit skip (api-security-3, audit 2026-08-28).
The ADR-0050 lake-route LB probe (#119) was added after
isUnauthenticatedInfraPath / SkipHealthAndMetrics were written and
missed both: under apikey / sep10 auth mode every uncredentialed
probe 401'd (contradicting the OpenAPI security: [] declaration),
and under apikey_optional it spent the anonymous per-IP bucket. The
exact-match lists and their pinning tests now carry the path. - Outlier filter trimmed agreed price moves; `outlier_storm` measured
its own artifact. The published-VWAP filter scored every print
against ONE band — the whole window's median ± 4 × 1.4826 × MAD — and
MAD is the *majority* regime's dispersion (0.1–0.3 % on a liquid
pair). Any agreed move larger than ~1 % was trimmed wholesale until it
became the window majority, then the old regime was trimmed instead.
Live on r1 (2026-08-28): Kraken stepped XLM/GBP 0.1337 → 0.1364, a
genuine +2 % matching the XLM/USD × GBP/USD cross; the window VWAPs
stayed pinned at the stale level,
anomaly_freeze_engaged fired on
the discontinuity when the regime flipped (sources=1 z=11.69), and
stellarindex_aggregator_outlier_storm fired for hours on
XLM/USD and XLM/GBP while every venue agreed within 0.9 % — its
counter re-counted the trimmed window tail every 30 s tick.
The orchestrator's filter is now time-local
(aggregate.FilterOutliersLocal): a print is dropped only when it
disagrees with the whole window AND its neighbourhood (own / adjacent
1-minute buckets when they hold ≥ 3 prices, else the nearest 5 prints
on each side — so thin single-source series are covered too). The
local references are anchored: their band is clamped to
0.25 %–1 % of the centre and a reference is trusted only when its
centre sits within ±4 % (σ × max(window scale, 1 %)) of the window
median or of the previous trusted reference (chain continuity) —
without this ANY burst that was the majority of its own minute
(≥ 3 prints) set its own centre and validated itself at any density;
the 2026-08-14 token-farm fixture passed 480/480 wave prints through
the unanchored prototype. Steps survive; a lone fat-finger, wash, or
dust print and a self-consistent 2–3× burst do not. Residual gap: a
wave that is the majority of the whole window, sits within ~4 % of
the honest level, or walks in ≤ 4 % steps is indistinguishable from
a market and still passes. The per-request /v1/vwap?outlier_sigma=
and /v1/ohlc filter keeps its documented whole-window semantics.
Regression tests replay the r1 shape (thin single-source +2 % step:
0 trimmed, z ≤ σ; lone +10 % print: trimmed) at the filter and
orchestrator layers, plus spam-bucket fixtures (dense single-venue
wash burst, mixed-venue spam burst, the 2026-08-14 token-farm wave:
480/480 dropped, 0 honest lost) and a permanent differential test
(survivor set + VWAP byte-identical to the whole-window filter on
normal / fat-finger / zero-MAD / tight series; legacy survivors ⊆
local survivors on 200 random series).
The alert now measures what it claims: new gauges
stellarindex_aggregator_venue_vwap{pair,window,source} (per-venue
pre-filter VWAP, absent venues deleted) and
stellarindex_aggregator_window_trades{pair,window,stage};
outlier_storm fires on max/min − 1 > 1 % across ≥ 2 venues for
15 m, and the new stellarindex_aggregator_outlier_trim_fraction
(> 20 % of the 24h window trimmed for 30 m, ≥ 20 trades) covers the
single-venue spam shape that disagreement cannot see — proven
fireable on the token-farm fixture: its promtool case uses the
filter's real output on that fixture (window_trades{stage=class} =
1200, {stage=outlier} = 720, a 40 % share), not hand-typed gauges.
promtool tests: agreed cross-venue step silent, one venue +3 % fires,
single venue never fires. The old counter gate is kept for one week
as stellarindex_aggregator_outlier_trim_rate_legacy (same expr /
for, both rule trees, three promtool cases) as the live cross-check
— retire 2026-09-04; after that dropped_trades_total
{reason="outlier"} is diagnostic only.
Fixed
- An asset whose XLM market is stored with the XLM SAC as BASE was
invisible to every price path. r1 2026-08-28 17:42Z:
stellarindex_assets_popular_priceless=2 for CBIJ… ($730k/7d, 706
trades against the XLM SAC, source=aquarius) and CAUP7… (trades only
against CBIJ) — price_usd null, no withheld verdict. The aquarius
decoder writes SWAP direction (base = token_in) without
canonical.Orient, so a token bought with XLM lands in prices_1m as
(CAS3J…, token). The volume path already read both directions
(soroban_volume.go) — which is exactly why the asset had volume and
no price — but every PRICE path read the XLM leg base-side only:
- asset_vs_xlm* in both catalogue queries (base_asset = X AND
quote_asset IN (native, SAC)): now UNION an inverted arm
(base_asset IN (native, SAC) AND quote_asset = X, 1/vwap).
Base-side rows are preferred over inverted ones, so every asset that
already priced keeps byte-identical output; the inverted arm only
fills assets with no base-side row in the window.
- TransitiveUSDPrice.hop_usd resolved a hop only via base_asset =
hop, so the XLM SAC itself (XLM/USD is keyed base_asset='native')
and any hop whose own XLM market is SAC-as-base priced NULL and was
dropped. Now: hop IS XLM (either identity) → xlm_usd; plus the
inverted XLM arm.
- The tripwire's priced_direct had the same base-only shape AND
never contained the proxies themselves, so one_hop could never
route through the XLM SAC. Now seeds the proxy set and reads the
inverted arm; coverageQuoteProxies is composed from the resolver's
own lists so the two cannot drift.
- GetAssetBySlug's chosen CTE was FROM classic_assets only — the
listing spine gained a discovered_assets UNION in #220 but the
detail did not, so /v1/assets/{id} for a Soroban-native contract
depended entirely on the transitive fill. Now the same UNION (same
asset_volume_24h bound).
Integration test TestXLMSacAsBase_PriceableThroughEveryPath (SAC-as-
base fixture; CBIJ priced 0.10 direct + transitive via the SAC, CAUP7
0.20 one hop through CBIJ, tripwire silent for both, and a both-
directions classic asset proven byte-identical) fails on every path
pre-fix. TestProxyQuoteLists_Lockstep pins the four proxy lists and
the 8 inverted arms. Writer-side follow-up (aquarius writing canonical
orientation) is separate; the read side must handle the stored data
regardless. - …and its price-history series (the sparklines) were still empty for
such an asset. Follow-up to the above: the four series queries
(
GetAssetPriceHistory24h/7d and their *Batch twins) each carry an
asset_xlm_per_hour/_per_day CTE that read the XLM leg base-side
only, so an asset priced through the inverted arm had a headline
price_usd but price_history_24h/7d all-null. Each now UNIONs the
same inverted arm (base_asset IN (native, SAC) AND quote_asset = ANY
(aliases), 1/vwap, vwap > 0), base-side preferred per bucket
(inverted ordered ahead of alias priority and bucket DESC), so
every bucket that already had a base-side point is byte-identical and
the inverted arm only fills buckets with none. TestProxyQuoteLists_
Lockstep now also pins the 4 series arms (the three inline queries
were hoisted to package constants for it) and the SAC-as-base
integration fixture asserts a non-empty, correctly-valued series on
all four paths plus the per-bucket preference. - Trade sink: a shutdown that raced an in-flight steady-state batch
write lost the batch instead of draining it. The pipeline sibling
of the sorobanevents AsyncSink fix (#240).
persistWorker's ticker /
batch-full flush runs under the parent ctx, so a SIGTERM landing
mid-BatchInsertTrades cancelled the write; context.Canceled is
(correctly) not an infra fault, so the batch fell into per-row
isolation against the same dead ctx and every row — up to 200
already-accepted trades — was logged "abandoned on shutdown —
re-derive" while the worker's own bounded shutdown flush ran a moment
later with nothing to do. flushTradeBatch now returns the trades
the cancelled ctx left un-landed; the steady-state flush carries them
back into tradeBuf for flushShutdown to land under the shared
drainTimeout, and only the BOUNDED shutdown callers (whose deadline
is the drain budget) report an abandon as loss, via one
reportAbandonedTrades helper. Audited siblings NOT affected:
discovery AsyncSink (per-record Background ctx, Stop closes + drains
the channel), clickhouse LiveSink (Background-ctx flushes, failed
flush keeps its buffer), externalRetryBuffer (re-queues on ctx
error, finalDrain under a fresh ctx), statsflush (delta against a
retained snapshot, final flush under WithoutCancel), and the
customer-webhook worker (durable DB lease queue). Regression tests
TestPersistWorker_ShutdownRacingInFlightTradeFlush_RowsLandNotLost
+ TestFlushTradeBatch_CtxCancelledMidWrite_ReturnsWholeBatch, red on
the pre-fix code (landed 0, want 3; 3 rows counted lost). - Aggregator gap detector took r1's serving path down (2026-08-28
18:23Z: `api_error_rate_high`, 503s from statement timeouts, load
19.6, IO-bound).
pg_stat_statements pinned it on the detector's
density query SELECT COUNT(DISTINCT ledger) FROM soroban_events WHERE
ledger BETWEEN $1 AND $2 — 121 calls, mean 556 s, 18.7 h total on a
257 GB hypertable that has no index on ledger (0041 partitions and
compresses by ledger_close_time, so a ledger-only predicate excludes
nothing). Three amplifiers, three commits, no DDL:
1. Timeout inversion. CountDistinctLedgers SET a 2 h PG
statement_timeout (the ops/verify constant) under a 15-min Go
context, so every over-budget count outlived its context as an
orphaned backend and each cycle / restart stacked another. Both
detector queries now share gapDetectorStatementTimeoutMS (13 min,
pinned ≤ the Go budget by TestGapDetectorStatementTimeoutWithinGoBudget).
2. Count source. The soroban-events target's density numerator
now comes from the ledger_ingest_log census (COUNT(*) WHERE
soroban_event_count > 0, a PK range scan) via the new
GapDetectorTarget.DistinctLedgerCountSQL override; every other
target's statement is byte-identical (differential unit test +
Docker-Timescale integration test). Gauge, snapshot row and
/v1/diagnostics/ingestion density_pct are unchanged in shape;
semantic note (LCM census vs observed rows) in the metrics README.
3. Restart storm. The per-target schedule was in-process and the
first cycle ran immediately, so every deploy re-ran the 6 h-cadence
heavy scans. The schedule is now seeded from the persisted
gap-detector-scan cursor; a skipped target's last-success stamp
and gap gauges are re-emitted from persisted state so the _silent
and gap_detected alerts keep working across restarts.
soroban-events stays on its 6 h cadence (the scan is now cheap; the
remaining 13-min-bounded LAG gap scan is unchanged). The runbook
ingest-gap-detected.md now lists every detector target whose table
has no leading-ledger index — the class this incident belongs to. - AsyncSink lost the in-flight batch on shutdown (#240):
Stop()
cancelled a steady-state write mid-INSERT and counted the whole batch as
LOST instead of handing it to the drain — every deploy that caught a
soroban_events batch in flight silently dropped it. The interrupted rows
are now re-queued into drainOnStop and retried under DrainGrace;
red-proven and stress-tested 600× under -race. - galexie tip-lag parser (#234) understands range object names on the
new archive schema (was reporting a bogus lag).
- `cut-release.sh --yes` (#231): refuses non-TTY runs without
--yes/--dry-run instead of aborting silently on read EOF. - Backfill-only archive schema vars (#230):
galexie_backfill_*
variables render into galexie-backfill.toml only, so a testnet 64/1000
re-export can never touch the live galexie.toml manifest; testnet
pinned to 64 ledgers/object. - Bespoke-cache staleness tests are deterministic (#264): an injectable
clock replaces a 10 ms real-time horizon that flaked on loaded CI runners.
Fixed
- `/v1/assets` still 500'd after v0.47.1 — the fix had addressed one
column, not the class. v0.47.1 COALESCEd
slug; production moved
straight on to code:
``
before: Scan error on column index 0, name "slug"
after: Scan error on column index 2, name "code"
`
catalogue_assets' Soroban arm supplies slug, code AND
issuer_g_strkey as NULL — a contract asset has no issuer account and
no SEP-1 code — and AssetRow typed all three as plain string. Any
one of them fails the WHOLE request, not just its row. The v0.47.1
test passed throughout, because it asserted on the SQL text for the
single column that had been noticed; a test written from a symptom can
only confirm that symptom.
Now fixed at the scan, where the class lives: code and
issuer_g_strkey scan through sql.NullString to "", and slug
does too with a fallback to asset_id that mirrors its SQL COALESCE.
Empty string rather than the contract id for code/issuer, because a
Soroban asset genuinely has neither and substituting an id would state
something false; the wire is unaffected either way (code is
omitempty, and issuer is guarded by a non-empty check before its
pointer is taken).
The remaining Soroban-supplied columns were checked rather than
assumed: first_seen_ledger, last_seen_ledger and event_count are
NOT NULL in discovered_assets, with 0 NULLs across the 60
qualifying rows. That is the complete set.
The new test drives scanAssetRow through a scanner reproducing
database/sql`'s actual NULL rule, with every NULL-by-nature column
NULL — proven red against the pre-fix scan and green after.
Fixed
- `/v1/assets` returned HTTP 500 for `limit >= 150`. A regression
introduced by #220 and first deployed in v0.47.0 — #220 merged after
v0.46.1 was cut, so v0.47.0 was its first time in production.
``
scan asset: Scan error on column index 0, name "slug":
converting NULL to string is unsupported
`
#220 taught catalogue_assets to UNION classic_assets with
Soroban-native contract assets. A Soroban row has slug **and**
code both NULL by nature — no issuer account, no SEP-1 code — and
the projection was COALESCE(ca.slug, ca.code): two arms, both NULL
for exactly those rows. slug scans into a non-nullable Go string,
so the whole request failed.
That file's own comment asserted "every downstream JOIN keys on
asset_id only, so the NULLs cost nothing." True of the JOINs. Not
true of the projection.
The fix adds ca.asset_id as a third COALESCE arm — the *correct*
fallback rather than merely a non-null one, since it is already the
asset's URL segment (/assets/CAUP7NFA… resolves), so the slug a
caller receives is the slug that works. Applied to both renderings
that carry the projection.
**Why it hid.** The failure is order-dependent: only 60 Soroban rows
exist and limit=100 never reaches one, so the endpoint looked
healthy at the default page size while being broken for every larger
caller (100 → 200, 150 → 500, 500 → 500).
**What caught it.** The explorer build fetches limit=500 and
refuses to bake fallback HTML on a failed fetch, turning a silent API
regression into a hard, visible build failure. That guard earned its
keep — and it is the third time in this release cycle that checking
the *served* result beat trusting a green signal.
**Measured blast radius:** 14 × HTTP 500 against 7,062 × HTTP 200 in
the 20 minutes after deploy, and all 14 were self-inflicted (operator
diagnostics plus the build's own retries). No real user traffic hit
it: the explorer is a static site already served, and the default
page size was healthy throughout. The API error-rate alerts fired
correctly at 04:39Z once their for:` window elapsed — there was no
detection gap, contrary to an initial reading taken before that
window had passed.
Fixed
- Deploys had been silently skipping two binaries, and nothing could
see it. Measured on r1:
stellarindex-ops at v0.44.7 and
stellarindex-migrate at v0.28.1 while
indexer/aggregator/api/sla-probe were all at v0.46.1. Neither was
ever reported, because a deploy that never touches a binary still
exits 0.
This is F-1314 recurring. That audit (2026-05-13) found
stellarindex-sla-probe drifting for exactly this reason and fixed it
by adding one name to the deploy workflow's default list — the
instance, not the blind spot. Three months later the same hole
swallowed two more binaries.
Neither is a bystander. Thirteen units on r1 exec
stellarindex-ops, including the data-integrity gates
(verify-archive tier-a/b, archive-completeness, ch-schema-drift,
restore-drill). A stale gate validates the lake against retired
rules, so it PASSES when it should fail — a silent false negative,
the one direction a gate must never fail in. stellarindex-migrate is
worse: deploy-binary.yml runs stellarindex-migrate up (F-1220)
*before* any binary swap, so every deploy was applying that release's
migrations with a months-old golang-migrate wrapper.
- The transitive price was invisible in the UI. v0.46.1 made
two-hop pricing reachable and the API has served it correctly since —
measured on CAUP7:
/v1/assets → price_usd 7768.93, price_basis
"transitive". The asset page rendered a permanent — anyway.
/v1/price answers for DIRECT markets only, and AssetPathView
passed initialPrice={null} regardless, discarding a price it had
already fetched in the same response. The entire population the
feature was built for — Soroban-native assets with no direct USD or
XLM market — showed no price at all.
That is twice this feature shipped with nobody able to use it: once
inert in the API (v0.46.0), once invisible in the UI. Both times the
server-side check looked healthy.
- `price_basis` was missing from the OpenAPI enum. Added to the Go
API in v0.46.x but never to the spec, so the generated client type
read
price_basis?: "declared_peg" and TypeScript could not name the
transitive case at all.
Added
- Own-binary version-skew detection. A node_exporter textfile probe
(every 30 min) plus two alerts, a runbook and a catalog entry, closing
the class rather than the instance. It compares binaries against
each other, not against an expected version: the host has no
authoritative notion of the current release, and hardcoding one would
make the probe lie after every legitimate deploy. It therefore also
covers binaries added later, with no list to maintain.
Scoped to the release-managed set (one per
cmd/ dir). Testing it
against live r1 found three hand-built operator one-offs —
stellarindex-ops-ch (v0.21.3), -claimable and -sacfix (both
dev), all from July, referenced by no systemd unit. Folding those
into the comparison would pin the alert firing forever on a condition
no deploy can clear, and a permanently-firing alert is the same as no
alert; they are reported as managed="false" instead.
- Issuer AccountEntry auth flags are now persisted. New
stellarindex-ops issuer-flags job with a daily timer (05:47 UTC).
This is durability, not a new capability — the flags already resolve
at read time (enrichIssuerFromAccountState, 39 of the top 40 issuers
measured 2026-08-27). What that path cannot survive is a cold
account-state cache, where an issuer page renders "not yet resolved".
Migration 0023 created the columns for exactly this fallback and
nothing had ever filled them: 0 of 59,189.
Reads by key_xdr, not account_id —
stellar.ledger_entries_current is ORDER BY (entry_type, key_xdr),
and the measured difference on that table is 0.069s vs 5.18s. At
~59k issuers that is the difference between a job that finishes and
one that does not.
Changed
- The deploy workflow's default binary set now includes
stellarindex-ops and stellarindex-migrate. Both are already in
deploy-binary.yml's cli_binaries deny-list, so they get a
-version smoke test and no service restart. Ordering caveat,
deliberate and documented: migrate is swapped *after* the migration
step, so a new runner takes effect from the next deploy — it
converges and adds no risk, and moving the swap earlier is a
structural change to the migration path on a live money database that
wants its own reviewed change.
Fixed
- Transitive pricing was shipped INERT in v0.46.0. Verified on r1
after that deploy: all four binaries on v0.46.0, and
CAUP7NFA… still
served no price_usd. applyAssetRowToDetail early-returns on
sql.ErrNoRows, and a Soroban-native contract asset has no
classic_assets row *by definition* — that table requires a G-issuer.
The whole premise of transitive pricing is "the catalogue cannot reach
this asset", so the one branch the feature had to survive was the one
the fill sat behind. Both arms now route through a shared
fillTransitivePrice, with a regression test proven red against the
shipped bug.
Worth recording *why* this was more dangerous than an ordinary bug: the
same release taught the coverage tripwire to see the one-hop path, and
that CTE is quote-based — so stellarindex_assets_popular_priceless
went to 0 on deploy while the asset remained unpriced. The alert
clearing and the gap closing are independent outcomes; only the first
had happened. Checking the alert board would have reported success on
what was, in effect, a bypass.
Added
- Soroban-native assets are priceable through one substance-gated hop.
The catalogue prices the long tail through exactly two hard-coded shapes
(
direct_usd, asset_vs_xlm), both built on classic_assets, which is
structurally classic-only (issuer_g_strkey NOT NULL) — so no Soroban
contract asset could reach either, however deep its market. CAUP7NFA…
traded $71.8k over 6,418 trades in 7 days and served no price. Now
price(A) = vwap(A/hop) × price(hop), with both legs independently
substance-gated: a two-hop price inherits its weakest leg, and an
ungated intermediate could reprice everything quoted against it. Served
last (never overriding an observed price) and marked
price_basis: "transitive". - ECB standby for the fiat-FX feed.
massive is a paid feed and was
the only series in stellarindex_external_fx_last_quote_unix, so a
lapse or a 429 broke every fiat-quoted pair once the 7-day forex-snap
lookback expired. ECB is free, keyless and authoritative; rates are
rebased onto USD (usdRate(X) = eurRate(X) / eurRate(USD)).
fx_quotes.source and the metric label follow the feed that actually
served. - OHLC `2h` / `12h` / `3d` / `2w`. Closes the resolution gap against
stellar.expert. No backfill —
/v1/ohlc already re-buckets a finer
continuous aggregate at query time, and all four divide cleanly into
existing CAGGs.
Changed
- The priceless-coverage tripwire's
priced CTE now recognises the same
one-hop path the resolver serves — with the same substance floors
($1,000 / 20 buckets / 6h), grouped per (asset, hop). Without those
floors the extension was a bypass, not a fix: 955 assets became
"priced", of which USDMPOOL ($798/24h) and yHELIX ($296/24h) would have
gone silent while remaining genuinely unpriced. Gated, it adds 7.
Fixed
- Test-net builds no longer advertise themselves as mainnet:
/network
reported "Pubnet" on testnet, 13 stellar.expert links pointed at the
mainnet explorer, robots/sitemap/canonicals named the production
origin, and API_BASE_URL fell back to the mainnet API when
NEXT_PUBLIC_API_BASE_URL was unset. A CI guard now greps for these
literals. - Transaction links point at our own
/transactions/{hash} instead of a
third-party explorer; outbound cross-references consolidated into one
line offering stellar.expert and stellarchain.io (network-aware; the
latter is the only one that exists for futurenet). galexie-archive-fill, verify-archive-tier-b, archive-completeness
and pgbackrest-backup are omitted on networks where they cannot work.
Tier B was anchoring testnet ledgers against pubnet hashes, and
archive-completeness was fetching pubnet checkpoints into a test-net
store — both wrong-network data paths, not mere noise.config-assertions no longer fails on layers a host doesn't have
(ZFS on the no-ZFS lean VMs); promtail no longer ships to a Loki that
isn't installed.scripts/dev/verify.sh mirrors CI's lint-metric-refs self-test, and
its gitleaks working-tree scan no longer red-flags real ansible vault
files — both had made cut-release.sh unrunnable on an operator box.
Added
- Testnet / Futurenet support — a one-line `stellar.network` switch. The
indexer now runs correctly (without corrupting data) against Stellar
testnet and futurenet. Grounded in a cold adversarial hardcode audit +
an independent fix-verifier pass (2026-08-26). Pubnet behaviour is
byte-identical (every new default resolves to the old constant).
- Config:
stellar.soroban_genesis_ledger / stellar.movements_floor_ledger
(pubnet values, or genesis=1 on test nets) so the SEP-41 supply and
CAP-67 real-time movements feeds don't floor above the whole chain;
timescale.MovementsFloor() + canonical.NetworkPassphrase() install
seams resolve leaf-package reads to the configured network.
- Corruption guards: SacContractID is network-aware (was serving the
pubnet contract address on testnet /v1/assets); config validation
rejects a pubnet (core-live) history_archive_url on a non-pubnet
network; the cross-anchor archive filler refuses to write pubnet
ledgers into a test-net archive; the SEP-41 supply genesis seed
defaults its boundary from the config's network value.
- Ansible: the archival-node role is network-aware (single-source
stellar_passphrase — fixes the futurenet core.cfg bug — per-network
history archive, boundary knobs, cap67 -floor-ledger); testnet +
futurenet inventory templates.
- CI/CD: deploy.yml gains testnet / futurenet targets; a fleet-model
design proposal (docs/operations/cicd-fleet-model.md).
- Docs: testnet/futurenet deployment guide + reset runbook.
Changed
- Real-time movement latency cadence tuning (~4s → ~2s). An adversarial
audit of a proposed captive-core "fast lane" found the live-movement latency
is a chain of hardcoded cadence constants, not a compute floor — so no new
component/second core is needed. Tuned the safe ones: the indexer's caught-up
MinIO re-check (
liveTailRetryWait) 3s → 500ms (the single largest term — a
caught-up indexer sat a flat ~3s behind the tip; MinIO is local, so a
re-check is a cheap bucket LIST); the /v1/ledger/stream poll that drives the
explorer's "watch it land" refetch 2s → 500ms; and the cap67 movements derive
tick (FOLLOW_INTERVAL) 2s → 1s. The ClickHouse-part-sensitive LiveSink flush
is deliberately left at 1s (sub-second flushing multiplies small parts on the
capacity-bound store). An event-driven MinIO-bucket-notification ingest (still
a single captive-core) is the documented next step toward ~100ms.
Added
- Real-time account-movements follow daemon (5.3). The CAP-67 movements
derive (
stellar.account_movements — the money trail served on
/v1/accounts/{g}/movements) now runs as a continuous follow daemon
(ch-cap67-movements -follow) instead of a ~30s timer + oneshot: it catches
up to the CONTIGUOUS lake tip, sleeps a short interval (2s), and repeats —
cutting movement latency from ~30s to ~2s behind the chain tip so a user
watches their transactions land in near real time. Builds on the contiguity
gate (#174, Cap67Range) so it never derives past a near-tip lake hole; the
timer is retired (single writer, no watermark race); Restart=always +
StartLimitBurst make a crash-loop trip to failed (visible to
node-healthcheck.sh, which now covers the daemon); a transient ClickHouse
error holds the watermark and retries (no ledger skipped). The initial
P23→tip backfill runs as the daemon's first catch-up. Operator cutover: apply
the ansible (or manually stop+disable cap67-movements.timer and start the
-follow daemon) AFTER the ops binary is deployed.
Added
- Soroban resource metering on `stellar.transactions`. Nine additive
DEFAULT 0 columns capture, per Soroban transaction, the DECLARED resource
bid (instruction count, disk-read / write bytes, read/write footprint entry
counts, total resource-fee bid) decoded from the tx envelope's
SorobanTransactionData, plus the ACTUAL charged fees (non-refundable,
refundable, rent) from the tx meta's SorobanTransactionMetaExtV1. Both are
decoded at ingest from the LedgerCloseMeta the indexer already holds; the
decoder is envelope-type-aware (unwraps a fee-bump to its inner tx — a naive
access nil-panics) and meta-version-aware (V3 + p27 V4). No
actual-instructions value is stored — pubnet ledger meta carries none (it
lives only in diagnostic-event core_metrics the lake does not store).
Populated go-forward; the sparse Soroban-only columns compress to
near-nothing. Requires the additive transactions_soroban_metering.sql
migration applied BEFORE the indexer binary (else the tx INSERT halts
ingest). Follow-up exposes the columns on GET /v1/tx.
Fixed
- `stellarindex_aggregator_outlier_storm` alert rescoped from a
self-poisoning relative-spike comparator (
>5× a [1h] offset 1h baseline —
a sustained storm's own drops entered that baseline window and flipped the
ratio false at ~72m, so the alert could never fire on the very storm it
exists to catch, and it ticketed on every benign single-pair robust-VWAP
trimming burst) to an absolute per-pair sustained gate
(sum by (pair) rate[10m] > 10 for 2h). Silent on transient dispersion,
fires on a persistent dispersion / broken-connector storm. - Explorer static export now rides out a transient 502/503/504 from the
API (typically the API mid-deploy) with the same patient, Retry-After-aware,
bounded wait
buildFetch already used for 429, instead of failing the whole
next build on one asset's momentary unavailability.
Fixed
- The
trades.signer sweeper (v0.44.4) failed every tick with tuple
decompression limit exceeded (SQLSTATE 53400) and tagged nothing: its
TagTradesSigner UPDATE joined on (ledger, tx_hash) with no ts
predicate, so on the ts-partitioned trades hypertable it scanned every
chunk — including compressed ones — and tripped the per-DML decompression
limit. Added the ts bound (the close-time span of the tagged txs, threaded
from the lake read) so TimescaleDB prunes to the window's chunks, mirroring
TagTradesRoutedVia. Same fix applies to the tag-signer backfill.
Tested against Stellar protocol 22.
Added
- AMM/Soroban swap actor attribution (`trades.signer`). The AMM decoders
(comet/soroswap/aquarius/phoenix) set
taker to the on-chain caller and
leave maker empty, so a router- or contract-driven swap had no
human/EOA attribution — the taker is the router contract. The tx source
account is that missing initiator, but it is NOT re-derivable from the lake
events the projector replays (they carry no source account), so it cannot
be set on the decode path. New nullable trades.signer column (migration
0150, mirroring routed_via: O(1) ADD COLUMN, deliberately kept out of the
trades UPSERT so a re-derive cannot clobber it) is back-tagged first-wins by
a trailing-window sweeper (pipeline.RunSignerTagger) that reads the signer
from the lake's stellar.transactions. The lake read is scoped to the small
ledger span of AMM trades still needing a signer (not every recent tx), so
it stays cheap at pubnet volume, and a per-tick ledger cap bounds a
cold-start / catch-up sweep. For gaps longer than the sweeper's 30-min
lookback (an indexer/ClickHouse outage or a projector lag), the
stellarindex-ops tag-signer -from N -to N command back-fills the range
through the same first-wins primitive. Exposed as signer on
GET /v1/accounts/{id}/trades.
Added
- /tx now shows the Soroban authorization-invocation tree. An
InvokeHostFunction operation's decoded fields gain an authorizations
tree — the nested SorobanAuthorizedInvocation structure from the op's auth
entries (contract + function + args, recursively), rendered on the explorer
/tx view as "Authorized invocations." This surfaces the nested contract-call
structure the view previously omitted (a step toward the richer /tx detail
stellar.expert shows). Decoded from the already-stored operation BodyXDR,
so no schema change or backfill. It is the AUTHORIZATION subtree, not the
full execution trace (that lives in the tx meta the lake does not store) —
labeled as such. The full execution tree + Soroban resource metering remain
a separate follow-up (they need tx-meta/resources the lake does not persist). - Exploit-shaped detector for AMM self-pair swaps (post-2026-08-25
Blend/Comet). A self-pair swap (
token_in == token_out) on a curated AMM
pool moves no value between distinct assets and has no honest purpose — it
is the primitive the exploit ran ~390 times to walk a pool's spot price,
and the freeze + divergence guards were blind to it because the self-pair
rows decode to zero rows and never reach the served trades table. New
counter stellarindex_amm_self_pair_swap_total{source} is incremented at
the comet decoder's drop point, and a stellarindex_amm_self_pair_swap_burst
alert (ticket) fires on increase[15m] > 10 — far above the historical-zero
baseline. Detection only: it changes no serving or freeze decision, so it
cannot create a false freeze. The counter increments only for LIVE (recent
ledger close time) events, so a backfill or completeness re-derive of the
historical exploit window does not re-fire the alert. Zero-seeded (F-0033)
so operators can tell "armed" from "dead metric," and ships with
false-positive + replay-suppression guard tests plus a runbook.
Fixed
stellarindex_priceless_coverage_check_stale paged a perma-stale FALSE
positive from the indexer and api instances. The
..._last_success_unix gauge is registered in the shared obs registry, so
every binary exports it, but ONLY the aggregator runs the coverage sweep
that sets it — on the other two it sits at unix 0 forever, so time() - 0
crossed the 1800s staleness threshold on every evaluation. Scoped the alert
expr to job="stellarindex-aggregator"; a genuinely-wedged aggregator
(its own gauge stuck at 0) still fires, and a promtool case guards the
non-aggregator-instance suppression.
Added
- Scam-pricing gate. An asset whose issuer is flagged scam-class
(
malicious/unsafe/fraud/scam/hack/phishing) in the curated
account directory now has its aggregated price AND market cap/FDV
withheld — a scam token no longer publishes a value that lends it
legitimacy, even when its market clears the thin-market substance floor
(RIO-GBNLJIYH… did: it showed a $0.0072 price + a $540k market cap on a
deprecated-scam issuer). Wired at the price-reader seam so one gate
covers /v1/price, /v1/price/batch, /v1/twap, /v1/vwap, the SEP-40
oracle price paths, the asset headline and the live tip (keyed on the
base, so it holds across quotes incl. XLM triangulation), plus a payload
suppression on the /v1/assets listing + detail (market_cap / fdv /
price / change). Raw trade surfaces (/v1/ohlc, /v1/observations,
/v1/history) and circulating_supply stay visible; the gate fails
open on a directory outage. Deliberately overturns the directory's
historical "display-only, tags never gate pricing" invariant. (#182)
Fixed
completeness_incomplete{source=comet} fired persistently after the
2026-08-25 Blend/Comet exploit — NOT a data gap (the lake is 100%
complete, watermark at tip) but a verdict artifact: the exploit's 36
self-pair swaps (token_in == token_out) fail canonical.NewPair, so
the comet decoder returned an error and the completeness re-derive
counted each as an undecodable blind spot, holding the source
complete=false forever (the INV-3 do-nothing re-derive trap). The
decoder now returns "zero rows, no error" for determinate business-rule
rejections (self-pair swap, non-positive amounts) so the re-derive
counts them as expected=0; the error path stays reserved for
indeterminate parse failures. (#185)external_fx_rate_rejections{reason=history_deviation} paged
indefinitely on a correctly-refused broken ETB history bar
(2026-08-19 = 44, the pre-float peg vs the correct ~160). The band was
right to refuse it; the _stuck reclassification that de-noises the
alert never engaged because the in-band branch cleared the stuck streak
that a good sibling bar in the same trailing-7d sweep had just
incremented. Removed the reset so a persistently-broken bar reaches the
_stuck threshold (and drops out of the alert), while a genuinely new
bad feed still pages. (#185)- Explorer asset-page price chart was blank for USDC and every fiat
currency: those chart against
fiat:USD, which the /v1/ohlc candle
path has no rows for (a fiat pair has no on-chain constituent; only
/v1/chart carries the fx-cross series). Fiat currencies now render a
USD line from /v1/chart, and USDC — the dollar reference, which has no
USDC/USD series of its own — shows a "≈ $1.00 reference" panel (linking
the divergence board for depeg watching) instead of an empty grid. (#184) - Explorer "Top assets by activity" ranked by all-time
observation_count
(a cumulative counter that floats long-lived stablecoins to the top) and
omitted native XLM entirely (it has no classic_assets row, so it never
appears in /v1/assets) — so USDC ranked #1 and XLM, the most-traded
asset on Stellar, was absent. Now ranks by trailing-24h volume and injects
native XLM (useNativeCoin over /v1/assets/native); XLM takes the #1
spot it earns on volume (~$43M vs USDC's ~$36M). (#181) - Explorer nav: restored a top-level Ledgers entry in the Stellar
section (it had been folded into the Network hub, making it undiscoverable
from the rail). (#181)
/v1/protocols/{name} per-contract activity ran the raw
contract_events FINAL scan (merge-on-read of the 12.8B-row
ReplacingMergeTree), which blew ClickHouse's 2 GiB per-query memory
limit (Code 241) — that memory kill *was* the "certified-lake reader
unavailable" verdict on the protocol page, and the 57s / 3.2B-row scans
were a primary CH-load source behind the API p95/p99 latency alerts and
the tx-outcome read timeouts. Now routed through the existing
contract_events_daily pre-aggregation (like the daily-activity and
event-breakdown views already are) — measured 0.5s vs 57s, no memory
kill. Last-seen is day-grain (sufficient for the roster column). (#180)/v1/anomalies returned 500 on every request — FreezeReasonCounts
and FreezeDailyReasonCounts used the same fragile ($1 || ' days')
interval concat that took down /v1/divergence: it types $1 as
text, but the handler passes an int, and pgx v5 has no int→text
encode plan. Also fixed the latent same bug in ListDivergenceSeries
(/v1/divergence/series, $4/$5). All now use make_interval, and
a package-wide test forbids the concat form so it can't return a third
time. (#179)
Added
- Live account-movements feed: the
/accounts/{id} movements view now
auto-follows the ledger tip — an SSE-nudged refetch (~4s coalesced,
shared tab-wide) gated to the first page so a keyset walk into history
is never yanked back, with a 20s fallback poll if the stream drops.
Classic-asset movement lag is cut from ~5min to ~30s by tightening the
incremental cap67 derive timer (the derive runs in ~0s per fire; true
up-to-the-second via a continuous follow-worker is a follow-on). (#172)
Fixed
- cap67 account-movements derive could permanently lose movements
under lake pressure. The
LiveSink drops whole ledgers under buffer
pressure, leaving holes near the tip, but the derive resolved its upper
bound with raw MaxLedger and advanced its watermark past any hole
with no trailing re-derive — so a dropped ledger's classic/native
account movements were never revisited (the raw lake self-heals via
ch-live-catchup; this derive did not → a permanent gap in account
history). Now clamped to ContiguousWatermark (mirroring the real-time
projector), so the derive stalls at a hole until catch-up heals it —
delayed, never lost. Found by an adversarial audit of the
real-time-movements plan. (#174) /v1/divergence (the per-reference divergence board) returned 500 on
every request — ListDivergenceLatest wrote its trailing-window
filter as now() - ($1 || ' days')::interval, which makes Postgres
infer $1 as text, but the handler passes sinceDays as an int.
pgx v5 has no int→text encode plan, so the query failed before
executing (unable to encode 7 into text format for text (OID 25)).
The window is now make_interval(days => $1), which types $1 as an
integer. Adds a regression test that forbids the text-concat form. (#176)- The
volume_character rollup worker refreshed the all-asset 14-day
account-structure roll every 15 minutes; each pass is a multi-minute
full scan of the trades hypertable (72M rows/7d), so it ran
effectively continuously and starved the customer API (the p99 2259ms
regression introduced in v0.44.0). The cadence is now 6h and the roll
caps its own max_parallel_workers_per_gather + statement_timeout so
a single refresh can't monopolize the primary. (#175) - Explorer pill contrast: category / venue / type chips now route through
the adaptive design tokens instead of hard-coded colors, fixing
low-contrast pills in the dark theme. (#173)
Added
- Materialized
volume_character rollup (design §2): a worker-maintained
per-asset table (migration 0149) computes the wash-vs-market
account-structure signals in one all-asset pass on the aggregate cadence.
The /v1/assets/{id} detail now reads it as a keyed lookup instead of a
per-request 14-day roll (which timed out at 4s on high-volume assets like
USDC), and volume_character is now carried on the /v1/assets listing.
(#35) - §4-B "annotate + demote": the default
volume_24h_usd_desc sort ranks by
concentration-adjusted volume (raw × (1 − top_account_pair_vol_share)
for concentrated/operational assets), so wash/operational volume no longer
tops the directory. The raw volume_24h_usd chain fact stays visible and
every asset stays present — a sort-key overlay only, never a value change
or a hidden row. (#35)
Security
- Bumped
golang.org/x/mod v0.39→v0.40 (CVE-2026-56864, CVE-2026-56865 —
malicious GOPROXY/GOSUMDB) and github.com/moby/go-archive v0.2→v0.3
(CVE-2026-17106 — tar path traversal). govulncheck clean. (#169)
Fixed
TestExternalFleet_EndToEnd integration flake: the consumer goroutine
inserted drained events with the fleet context that shutdown cancels
mid-drain; inserts now use a decoupled context. (#169)
CI / tooling
- Dependabot ignores TypeScript major bumps in the explorer (openapi-typescript
is not yet TS7-compatible), stopping a recurring red PR. (#169)
- Corrected the stale
ansible-drift comment: the vault secrets are restored
and the check works; a failure now signals genuine r1 drift. (#169)
Operator notes
- Migration 0149 (
asset_volume_character rollup table) applies via the
standard deploy migration step. No new Prometheus rules or systemd units.
Added
- Alias-aware asset directory: SAC/alias twins fold onto their canonical
classic row (summing 24h volume + trades with exact
big.Rat); a configured
SAC asset_id resolves to its classic on the detail path. (#28) - Priceless-popular pricing-coverage tripwire: an aggregator sweep pages when a
genuinely popular asset (market-character volume, wash excluded) has no served
price. New
stellarindex_assets_popular_priceless gauge + sweep-health
metrics, bounded by a 5m per-sweep timeout. (#28) - stellar-expert scam-label +
volume_character signals surfaced on the asset
directory + detail (issuer_directory_{tags,domain,name}, volume_character,
volume_character_signals). (#30) - Per-account activity watermark bounds the ops-by-account ClickHouse scan
(fail-safe: a missing watermark falls back to the pre-existing unbounded
scan). (#31)
- W8 ops observability: projector-wedge gauge, notify-send metric, and a
verify-archive Tier-B nightly timer. (#33)
Fixed
/v1/network/stats stamps honest flags.stale + as_of on the
stale-while-revalidate serve path instead of silently asserting fresh (REC-05,
same class as /v1/markets)./v1/markets stamps honest stale + as_of on SWR stale-serve. (#160)- Convert page hydrates header/inverse/ladder live off the shared query instead
of serving build-frozen residue. (#32.10a)
Security
- Removed permissionless DeFindex strategy self-registration: the curated
MainnetStrategies set is the sole trust root; a factory create body can no
longer seed a poisoned strategy into the gated registry. (W8 6c)
Operator notes
- New Prometheus alert rules (
notify, pricing-coverage, projector,
verify-archive), the verify-archive-tier-b systemd timer, and the
account_activity ClickHouse table + MVs are config/schema that a
binary-only deploy does not apply — apply them alongside the binaries.
Added
- Declared fiat-peg pricing (AUDD/AUDR → AUD × served fx), price_basis=declared_peg. (#154)
- USDC + SAC quote bridges for directory pricing. (#152)
Fixed
- FX confirm-veto: an agreeing 7d history majority refuses a pending confirm, stopping the persistent-broken-upstream (UZS) re-poison; genuine devaluations still confirm. Outlier-drop counter gains a pair label. (#157)
- /v1/assets/{id} detail overlay is substance-gated; dust prices no longer leak onto detail. (#154)
- Account ops pages: detached budget for the tx-outcome stitch. (#155)
Fixed
- FX guard: history-majority confirm veto (the Massive UZS second
act). A persistently-broken current feed can no longer re-poison a
healed baseline through the two-fetch confirmation: when a ticker's
trailing-7d majority (≥4 bars mutually agreeing within 10%) REFUTES a
pending candidate, the confirm is refused
(
deviation_history_conflict; repeats reclassify to
…_conflict_stuck, excluded from the rejection alert). History still
never SETS a baseline — genuine devaluations confirm as soon as the
majority stops refuting (follows the move, or the split window yields
no majority). Red-proven tests. (task #29)
Added
stellarindex_aggregator_dropped_trades_total now carries a pair
label (the configured target pair, bounded ~12), so an outlier_storm
is attributable with topk by pair instead of ad-hoc SQL — the
2026-08-14 single-issuer SDEX token-farm wave took the latter.
Storm/spike alert exprs sum() across labels and are unchanged.
(task #29)
Fixed
- FX guard: history-majority heal for poisoned bootstrap baselines +
jitter-tolerant stuck streak (the Massive UZS incident). A broken
current-feed bar bootstrap-accepted at restart no longer poisons the
baseline against the ticker's own correct 7-day history: ≥4
mutually-agreeing rejected bars refute an unconfirmed single-sample
baseline (median wins, the poisoned current-day row is scrubbed
before write). Confirmed baselines are never healed; split series
never heal. Stuck-streak reclassification now tolerance-matches
(exact float equality never matched a live jittering upstream).
Two-lens verified; red-proven tests. (#146)
Added
- Synthetic USD-cross divergence reference — non-USD-fiat pairs
(XLM/EUR, XLM/GBP, …) get a second reference (on-chain oracle
base/USD ÷ reflector-fx or chainlink fiat/USD), so SuccessCount
reaches the divergence trust floor and ADR-0019's corroborated
release can auto-release genuine repricings unattended — four
operator freeze-releases on 2026-08-24 alone were this class.
Migration 0148 admits the source to divergence_observations
(pure-widening CHECK; decompress dance). Two-lens verified incl.
live migration exercise against compressed chunks. (#149)
Fixed
- Freeze lifecycle: escalated-freeze ratchet + restart stall + corroborated
release (ADR-0019 amendment 2026-08-24). Mid-freeze buckets now score
per-tick returns against a shadow comparator (kills the drift-since-freeze
ratchet that kept XLM/GBP-style freezes from ever releasing at a new stable
level, and the restart→unscored stall). Because any HELD level reads calm
under per-tick scoring, auto-unfreeze now additionally requires a
corroborating lens reading that agrees within 5% with the bucket's own fresh
candidate price (
Signal.ReleaseCorroborated): a genuine repricing whose
references follow releases; a parked manipulation walks the ladder to the
operator. Pairs with no usable reference never auto-release (fail-closed;
they escalate and page). Verified by a 3-lens adversarial panel; red-proven
regression tests at both the policy and orchestrator layers. (#142)
Changed
- Explorer: /sdex is the one canonical SDEX surface (protocol analytics view;
/protocols/sdex 301s server-side). Accounts page frame + logo polish.
(#141, #143, #144)
Fixed
- /v1/price p95 tail eliminated: the serving pool now forces custom
plans (
plan_cache_mode=force_custom_plan post-connect). Root cause:
Postgres flipped the request path's raw-trades fallback to a generic
plan whose build costs ~206 ms across the ~870-chunk trades hypertable
and is rebuilt on every plancache invalidation (~1/min) — a steady
~5 % of serving binds paid 250–330 ms. Custom plans bind in 0.2–3 ms.
Background/ops pools keep the default plan mode. - Stuck-upstream FX rejections no longer hold the alert red: after 12
consecutive refusals of the SAME broken history bar (the Massive
ETB=44 case) repeats reclassify to
history_deviation_stuck, excluded
from the alert; fresh disagreement still alerts immediately. The guard
refuses the bar either way. - Explorer: navigation revised into Stellar / External / Developers
sections, Stellar-mark + Inter wordmark logo with the live ledger
number beside it, new /insights hub, /network sub-surface links.
No migrations.
Fixed
- OHLC bars are now bit-for-bit reproducible (migration 0147): the price
CAGGs' open/close resolve same-instant ties by a total key
(epoch-µs ‖ ledger ‖ tx_hash ‖ op_index ‖ source) mirroring the raw-trades
serve order, instead of physical scan order. VWAP switches to the exact
single-division form (≤1e-16 relative, below wire truncation — the
0115-invited free rider). ⚠ The migration recreates the seven price CAGGs +
twap_1h/1d WITH NO DATA; re-materialization is the deploy follow-up
(recent-first plan in the migration header).
- Freeze markers now write for Phase-2 freezes on Phase-1-off deployments:
the freeze writer was gated on the Phase 1 anomaly checker while the Phase 2
confidence lifecycle runs unconditionally — engaged freezes (r1 XLM/GBP)
refused publication with no Redis marker, serving the last value with
flags.frozen absent. Writer is now built unconditionally; AST tripwire
added. - The daily supply-snapshot writer can now actually run: the auto
snapshot-ledger resolver clamps to the lake's landed tip (bounded, 512
ledgers) instead of demanding the realtime cursor's not-yet-landed
stellar.ledgers row — the structural race that failed every timed run.
Operator
-ledger stays exact fail-closed; wall-clock stamping remains
impossible. - Integration tests quiesce CAGG refresh policies in the shared bootstrap
(the 55P03 concurrent-refresh flake).
Changed
- Monitoring: the system recognition census is a drift gauge
(
stellarindex_recognition_unattributed_shapes) with a step-change alert,
no longer a permanently-red completeness_incomplete row; new
galexie-archive partition-contiguity guard (hourly scan + page alert on any
gap/overlap outside the declared capacity trim). - Explorer: shared LastPriceCell (restores the tick flash DexesView's fork had
lost), /dexes pools board follows ledger closes, home Recent Trades ticks on
ledger closes instead of a blind 30s poll.
Added
- `GET /v1/livez/lake` — the lake-critical LB probe (ADR-0050 §7.3):
200 iff ClickHouse pings; 503 on failure or when no lake is wired
(fail-closed). Complements
/v1/readyz's deliberate CH-non-criticality
so a lake-dead instance can be pulled for lake routes without touching
pricing. - SLO lake-guard test: CI now fails if any SLO'd handler
(
/v1/price*, /v1/oracle/*) reads a ClickHouse-backed field — the
"no SLO'd route touches the lake" invariant, enforced.
Changed
- Phoenix completeness reconcile is now STRICT per-ledger — the
aggregate netting opt-out is retired (own-ledger attribution removed
the sweep-shift it absorbed; proven with 0 mismatched ledgers before
removal). A real drop can no longer net against a phantom.
Fixed
- Served-reader determinism:
TradesInRange gains the full ORDER BY
tiebreak (raw OHLC bars no longer depend on arbitrary same-timestamp
ordering); account_movements gains the LIMIT 1 BY read-time dedup
its sibling readers already had; NetworkThroughput derives its day
window and Partial flag from the data's tip close time instead of the
wall clock.
Added
- DeFindex `dfees` fee distributions are now modelled (W5.2, the last
open launch item). Body shape proven from captured on-chain blobs
(
Map{"distributed_fees" → Vec[(token, i128)]}, per-asset, empty vec
valid): one row per distributed-fee token into the new defindex_fees
table, with full sink/projector/reconcile registry parity. The ~12.8K
historical events backfill via projected-rebuild -source defindex
after this release deploys. - Explorer feels alive: live data across the whole site. Pool
reserves, pair tables, and lending reserves refresh on every ledger
close (shared
useLedgerFollow); charts advance their forming candle;
the home "live USD price" actually streams and flashes; venue/DEX
last-price cells flash again; the asset History tab is a true live
trade tape over the previously-unused /v1/observations/stream;
rollup panels and activity feeds auto-poll. - Failed transactions are first-class on the explorer and API with
explicit failed status and failure reason (D-PART-FAILEDTX decision),
plus the 2026-08-14 audit's decisions batch.
- OpenAPI spec overhaul: exact route parity (129/129 with unique
operationIds), valid OpenAPI 3.1 null unions, 0 Spectral errors,
regenerated Postman + types.
Fixed
- Completeness re-derive counts sweep-rescued outputs at their own
ledger (
eventLedgerCarrier): a correlation-buffer rescue (phoenix
7-field era) is now attributed where its served row lives instead of
at the sweep-trigger ledger, removing the CS-084 ± shift noise from
strict per-ledger reconciles. - `projected-rebuild` clamps `-workers` to 1 for correlation-buffer
decoders — concurrent out-of-order windows starve sweep triggers and
silently drop groups (measured: 4 workers lost ~650 of 5,154 phoenix
era trades in a dry-run; 1 worker lost none).
- SQLSTATE class extraction guards malformed codes (
sqlStateClass)
instead of slicing blind. - The pgBackRest restore drill had never once run on its schedule
(BDR-04). CS-110's whole point is evidence that the backups restore,
and the scheduled path produced none — for three stacked reasons, each
hidden behind the one before it:
1.
PrivateTmp=true gives the unit its own empty /tmp and /var/tmp,
so DRILL_ROOT=/var/tmp/restore-drill — a provisioned 5.2 TB ZFS
dataset, plainly present on the host — did not exist inside the
service's mount namespace. ReadWritePaths on that path failed
namespace setup and systemd aborted the unit with 226/NAMESPACE
BEFORE ExecStart. Every passing drill on record was run by hand,
which has no namespace.
2. With that cleared, NoNewPrivileges=true blocked sudo's setuid
transition ("unable to open /etc/sudoers: Operation not permitted").
The unit runs as root by design and DROPS privilege to postgres;
no-new-privs protects nothing on an already-root unit while
disabling the one mechanism it uses to run with LESS privilege.
3. Then pgbackrest, running as postgres, could not traverse
/var/lib/stellarindex (drwxr-x---).
The dataset now lives at /srv/restore-drill, postgres-owned — /srv
is world-traversable, is not shadowed by PrivateTmp, stays writable
under ProtectSystem=full, and already hosts history-archive. The
ZFS role gained optional per-dataset dir_owner/dir_group
(default(omit), so every other dataset is untouched). - `tip_lag` was measuring the backup's AGE, not recoverability (BDR-05).
The scratch instance runs
hot_standby = on and is started with
pg_ctl -w, which returns the moment CONSISTENCY is reached — while
replay of the remaining archived WAL continues in the background. The
drill then measured the restored tip immediately, so the number it
reported was "how old was the backup we restored from". Measured
2026-08-19: lag 13,392 ledgers (~18.6h) against a diff taken 21h
earlier, while archive-get was demonstrably still streaming segments
in ~10ms each minutes later. On a daily-diff schedule that made the
< 5000 threshold unpassable except by drilling shortly after a diff —
the 2026-07-03 pass (240 ledgers) was exactly that accident, and a
threshold met only by luck is not evidence. The drill now drains the
archive stream to an LSN captured from the live primary before
measuring, treating BOTH terminal states as drained (replay passed the
target, or recovery ended and promoted — the latter returns NULL from
pg_last_wal_replay_lsn() and would otherwise spin to the timeout on
the very run that succeeded). The drain is a reported check of its own,
so a timeout can never masquerade as a clean measurement. - The
ReadWritePaths directive is gone entirely rather than repointed:
ProtectSystem=full already leaves /var and /srv writable, and the
directive's only effect here was to make a missing path a hard start
failure.
Fixed
- phoenix stake-init events no longer trip `recognition_ok=FALSE` (#108).
20 real
LP-share staking init events matched no decoder shape, so the
ADR-0033 recognition census counted them as unhandled topics and downgraded
the whole source — even though they carry no financial row to project. The
decoder now *recognises* the init topic (Matches() returns true) and
emits nothing by design, so recognition is honest and projection is
unchanged. No served-data change. - sorocredit `TreasuryUpdated` config event is now recognised and captured
(#108). The main contract's
TreasuryUpdated topic (a treasury-pointer
rotation, body Vec[Address old, Address new]) matched no decoder shape —
one real lake event at ledger 63,847,367 was dropped end-to-end, tripping
recognition_ok=FALSE. It is now captured verbatim into
credit_events.attributes["body"] (exactly like BeaconUpdated /
CollateralHashUpdated), with migration 0145 admitting treasury_updated
into the credit_events_event_type_check CHECK. No promoted column, no
invented semantics. - blend_emitter reconcile fan-out false-red (#107). The projection
reconcile compared served rows against a lake re-derive that counted the
drop event_kind — a fan-out kind the emitter carves out of the served
projection — inflating the expected count and reporting a phantom
Σ|Δ|=14 mismatch on a source whose data was always correct. The reconcile
now excludes drop (event_kind <> 'drop'), so blend_emitter_events
reconciles exactly. No served-data change.
Fixed
- Completeness reconcile no longer times out on factory-gated sources (#104).
The
-pass per-source projection re-derive streamed the entire ~6B-event CH lake
for identity-gated sources with empty catalogue contractIDs (aquarius, phoenix),
blowing the 120-min pass deadline (aquarius: projection: context deadline
exceeded failed the whole pass on r1). For opted-in gated sources it now scopes
the -ch re-derive to a guaranteed superset of the gated contract set (factory
∪ curated seed ∪ protocol_contracts children ∪ lake-announced children) via the
contract-indexed contractIDs prefilter — counts-identical to the full stream
(Matches() rejects non-gated contracts regardless), just orders of magnitude
faster. Opt-in is pinned to {aquarius, phoenix}; defindex is excluded (its decode
correlates events across contracts in a tx, which a contract prefilter would break).
Fail-closed: a missing contract would under-count → a visible red, never a false green.
Security
- govulncheck gated behind a documented lib/pq accepted-risk allowlist (#105).
The 2026-08 CVE-2026-56868..56874 batch surfaced 7 unpatched *called*
vulnerabilities in
github.com/lib/pq@v1.12.3 (the latest release of the now-
unmaintained driver), failing CI on every PR. All require a malicious/compromised
Postgres server or a pre-auth MITM; stellarindex connects only to its own Postgres
over 127.0.0.1 (sslmode=disable, no GSS/.pgpass) → not exploitable in this
deployment. A reviewed allowlist (scripts/ci/govulncheck-allow.txt + a JSON-mode
wrapper that still reds CI on any *other* called vuln) documents the accepted risk;
the durable fix (migrate to jackc/pgx) is tracked as a post-launch follow-up.
Fixed
- soroswap recognition false-red (#100). The ADR-0033 recognition census
built its dispatcher without the soroswap pair registry, so its soroswap
decoder's
pairTokens map was empty and Matches() rejected every real
SoroswapPair protocol event — each became a false "unhandled topic" gap
attributed to soroswap (and the watermark clamp cascaded into spurious
projection floor-loss alarms), even though the indexer decodes + serves those
trades correctly. Both recognition-census paths now seed the soroswap decoder
from the same LoadSoroswapPairRegistry set attribution already uses. - aquarius `set_protocol_fee` Vec-body decode (#101).
set_protocol_fee
events on registered Aquarius pools carry a Vec body (SCV_VEC[SCV_U32] = the
new pool-wide protocol-fee fraction, per the pool WASM's singular
set_protocol_fee_fraction) that the Map-only decoder dropped, blocking
aquarius projection with "undecodable-but-matched" blind spots. decodeFee now
branches on the SCVal kind; the absent prior fraction lands NULL (not invented). - phoenix incomplete gating seed (#102). The curated
MainnetGatedSet was
missing 14 verified-genuine phoenix contracts (1 pool + 13 per-pool stake
contracts), so the reconcile under-counted them AND the live gated pipeline was
silently dropping some still-active contracts' events. All 14 were verified
on-chain (factory pool-create co-occurrence / shared reward keeper / stake-v1.1
migration events) and added to the seed; the pre-upgrade 7-field sweep-emit
ledger shift is absorbed via aggregateReconcile. - defindex projection dirty window re-verified clean and cleared (the #91
harvest-count fix, live since v0.36.0).
Added
- Comprehensive per-source projection reconciliation + a static
catalogue-completeness invariant (#96). The projection axis previously
reconciled only a subset of protocol tables; it now carries reconTargets for
the 1:1 tables it was missing (aquarius admin / protocol-fee / kill-switches /
liquidity / reserves-sync / rewards, soroswap_liquidity, phoenix
initialize/admin), with the genuine per-token fan-out tables (aquarius
reserves/liquidity, sdex_offer_events) explicitly
noReconcile-waived rather
than left silently unvalidated. A new AST-walking invariant test asserts every
decoder EventKind that routes to a persisted table is either
reconciled-by-kind/census or explicitly waived — so a future decoder kind can
no longer silently fall out of the reconcile's EXPECTED sum (the exact class of
the defindex strategy.harvest undercount that produced a phantom
976-mismatch false-red).
Fixed
- The nightly completeness-verdict driver no longer times out and freezes the
alphabetical tail.
run-compute-completeness.sh re-invoked
compute-completeness -ch per source AND per 25k chunk, and every invocation
re-ran the load-heaviest step — the global DistinctTopicShapes recognition
scan (~60s over full history, identical regardless of -source/-from). A
source pinned far below tip (aquarius, recognition-capped near its genesis)
walked hundreds of chunks, so that one scan ran hundreds of times per night —
the 3h52m timeout (Result=timeout) that left the alphabetical tail's verdicts
days stale. A new compute-completeness -ch -pass mode proves recognition +
substrate ONCE at full range for the whole catalogue and reconciles each
source's projection incrementally from its own watermark; the wrapper now makes
one such call. This also (a) clears the low-tip substrate flap — a full-tip
substrate proof advances the tip and is never blocked by the CS-083 write guard
— and (b) finally gives every catalogue source a verdict, including the
never-seeded blend_emitter/blend_backstop/sorocredit (they reconcile from
genesis on the first pass). INV-5, the projection dirty-window mechanism, the
substrate/projection fail-closed claims and CS-083 are all preserved unchanged.
Fixed
- Aquarius pool governance events are no longer silently dropped. The decoder
gated 7 governance topic symbols (
apply_upgrade, commit_upgrade,
set_privileged_addrs, apply_/commit_transfer_ownership,
enable_/disable_emergency_mode) on the canonical router only — but the 337
registered Aquarius pools emit them too (a protocol-wide staged WASM upgrade of
320/337 pools). Pool-emitted governance events returned Matches()=false,
becoming an ADR-0033 recognition gap (holding aquarius completeness red) AND
never reaching Decode → ~1,679 real events lost since ledger 55,363,632. The
gate now accepts registered pools (reg.Has || reg.IsFactory; unidentified
emitters still fail closed), and the upgrade decoder handles the pool body
arities (router = 1 wasm hash, pool apply = 2, pool commit = 3 → staged
hashes in attributes.wasm_hash_N). The events now land in the already-served
aquarius_admin table. (A backfill re-processes the historical drop.) - Defindex `strategy.harvest` flows are counted in the completeness verdict.
The reconciliation catalogue omitted
defindex.strategy.harvest from the
defindex_flows expected-count kinds, so the ADR-0033 verdict under-counted
every genuine harvest by exactly 974 (served=1, expected=0), false-flagging
defindex complete=false. The served data was correct; adding the kind fixes
the count. Count-only — no data mutation.
Fixed
- XLM circulating-supply refresh no longer falsely freezes during quiet
periods. The supply freshness gate anchored on
MAX(ledger) over
account_observations, which only rows on a watched SDF-reserve-account
*balance change* — so any market-quiet stretch beyond the ~1-day dormancy
horizon made the anchor go stale and the gate fail closed, freezing XLM
supply and firing a continuous supply_refresh_error_dominant ticket (which
in turn masked a genuine future observer death). The served value was always
correct — only its freshness signal was wrong. The gate now anchors on a true
per-tick observer watermark (new account_observer_watermark table, migration
0144), advanced every ledger by the indexer: a healthy-but-quiet observer
stays fresh, a genuinely dead observer still trips the gate. Found only by a
live audit of r1 — the code looked correct; the live quiet-reserve state
triggered the latent flaw.
Security
- Account-history participant injection closed. A Soroban
InvokeContract
op's call arguments and SorobanAuthorizationEntry entries are attacker-
controllable at the XDR-decode layer, so they are no longer indexed as
account participants. Previously an attacker could inject an arbitrary
victim's address into that victim's permanent, public
/accounts/{g}/operations history under the attacker's own signature. - audit-2026-08-14 remediation — 79 verified fixes across money-correctness
(SDEX single-leg plausibility ceiling, oracle-execution corroboration for the
Band adapter, MEV-detector evidence-gating +
mev_events retention),
auth/data-integrity (self-service key-mint scope hardening, session token
hashing at rest via migration 0143), and projector durability. Each landed
with a proven-red regression test. - Go toolchain 1.25.12 → 1.25.13. govulncheck reported 7 standard-library
vulnerabilities reachable from live call paths —
net/http (GO-2026-5026,
Punycode label handling) via the ClickHouse reader, the CoinGecko supply
client, the history-archive checkpoint resolver and the galexie trim's S3
calls, and encoding/asn1 via the WebAuthn passkey registration path. All
are fixed in go1.25.13. Every workflow reads go-version-file: go.mod, so
the toolchain directive is the only pin to move. Verified clean locally:
"0 vulnerabilities".
Changed
- Asset identity: one alias registry. A binary-startup
AliasRegistry
built from [supply].sac_wrappers folds an asset's SAC-wrapped form into a
single identity (SAC form ordered last), threaded through the price/volume
read paths. Fixes alias-blind volume/price reads across ~11 money endpoints
(asset detail, VWAP/TWAP/OHLC, pairs, markets, aggregate global tiers) that
previously split an asset's SAC and classic forms into two un-aliased
identities. Non-XLM folding activates per [supply].sac_wrappers config.
Fixed
- Absent-vs-zero honesty across the read surface.
/v1/status incidents
now carry an explicit ok|degraded|unknown tri-state (a failed alert query
no longer serialises as a false all-clear); /v1/tx distinguishes partial
event / op-result coverage; /v1/protocols serves from an SWR cache instead
of a per-request unauthenticated scan; the explorer degraded-banner and
network-insight no longer read a failed query's zero as real data. - Incidents Atom feed `<updated>` now reflects the most-recent entry's
timestamp (empty feed → a stable sentinel) instead of wall-clock
now(), so
a stale or empty feed is no longer syndicated as freshly updated every crawl. - `TestMigrationsRoundTrip` could deadlock against TimescaleDB's own
job scheduler, turning `main` red for 30 hours and firing the
ci-health tripwire every two hours. The test asserts compression
and CAGG-refresh policies are attached, then rolls every migration
back — so
migrate down's DROP ... AccessExclusiveLock raced the
16 background workers running those very policies, and the two could
form a lock cycle ("deadlock detected, Process 94 waits for
AccessExclusiveLock on relation 21724; blocked by process 161"). It
only reproduces under load, which is why it passes locally in 5s.
Retrying is not available as a fix: a failed migration leaves
golang-migrate's version DIRTY. The container now runs with
timescaledb.max_background_workers=0, removing the concurrent
actor entirely, and the test asserts the setting actually applied —
a Cmd override that silently failed to take would otherwise look
exactly like a fix. The assertions are unchanged in strength: they
check policies are ATTACHED (a metadata row), not that they run.
Changed
- `GET /v1/contracts/{id}/interactions` now anchors its window to the
contract's own recent activity, so `?days=` is an UPPER bound rather
than the window served. Both halves of the read scale with the
ledger span they cover, and over the default 90 days a busy contract
cost 3–6s — the slowest panel left on the contract page once the
/wasm scan was bounded. Narrowing to the contract's 500 most recent
active ledgers brings that to 0.705s. This is a deliberate trade, not
a free win: shared_txs counts drop for busy contracts. The ranking
— which is what the panel is for — was unchanged in the same order on
the measured sample, and the endpoint has always reported a bounded
recent sample (subjectTxCap truncates at 50,000 transactions).
Quiet contracts, which are most of them, have fewer active ledgers
than the cap and keep the full window. since_ledger reports the
floor actually served, and the OpenAPI description now says so.
Fixed
- Every cold contract page served at least one failed panel, because
the page starved itself at the refresh gate. The contract view fans
out to five concurrent reads, but four of them (detail events,
interactions, code-history, account activity) all acquired the single
refresh-gate class
contract_detail, capped at half the global limit
— two slots. So on a cold contract two of the four refreshes were
refused, and a refusal with nothing cached is a 503, not a stale
serve. Measured on 20 of 20 cold random contract pages, and it was
not crawl pressure: the same rate held with seconds of think time
between pages. The per-class cap exists to stop one class starving
the OTHERS, so the classes are now keyed per panel, which restores
that intent without letting a page compete with itself. The global
bound was also below one page's width (4 for a 5-read page) — raised
to 8, with the explorer ClickHouse pool 8 → 16 so "detached refreshes
can never consume the whole pool" still holds. r1 has 20 cores and
idles at ~2 concurrent queries, and every explorer scan is pinned to
max_threads = 4. scripts/ops/contract-page-audit.py now scores a non-2xx/404 panel
as UNLOADED instead of as a fast response, and reports it separately
from latency. The first version counted a 503 as a loaded panel, so
it rated pages "ok" at 0.10s while three of five panels were failing
— a broken page scored better than a slow one. It also takes PACE,
because "is one cold page fast" and "does the site hold up under a
sustained crawl" are different questions and were being answered by
one number.
Security
- Registered API keys were completely unmetered in production
(audit 2026-08-13 F1):
MirroredKey carried no monthly quota, so the
record the deployed Redis validator reads had none, and the quota
middleware short-circuits at <= 0 — every key /v1/register handed
out advertised a 1,000,000/month cap (in its own response body and in
the public agent docs) and was enforced nowhere. The rate limiter was
the only live bound. Quota now flows through the mirror, with a
round-trip test (real store → real validator) asserting LITERAL
expected values: the prior tests compared a component against its own
input, which is why a dropped field read as correct on both sides. - `POST /v1/register` was cross-site invocable (F4): the
Content-Type gate only validated the header when present, so a
header-less POST — a CORS *simple* request, never preflighted —
let any page create an account plus a permanent credential per
visitor via
fetch(…, {mode:'no-cors'}), while burning tokens from
the per-IP throttle this endpoint shares with /v1/signup (with the
source addresses distributed across victims). The header is now
required; docs and examples send it.
Fixed
- A contract page took ~8s to finish loading because the WASM panel
paid an unbounded lake scan to produce a nicer 404. When a
contract has no captured instance — the common case —
/wasm asked
"is this a SAC?" via contract_id = ? ORDER BY ledger_seq DESC LIMIT
1 over contract_events, the quiet-contract reverse-scan trap that
contract_active_ledgers exists to prevent. That cost ~0.34s idle,
but the contract page fires five reads at once and the other four
return via stale-while-revalidate while spawning background
refreshes, so the inline WASM read was starved to its full 8s
request deadline. 23 of 25 cold random contract pages breached the
1s budget on this single call, and it also starved sibling panels
into intermittent 503s. The probe is now bounded to the contract's
own recent active ledgers (0.008s measured on r1, ~40x), and an
empty active-ledger walk answers authoritatively without touching
contract_events at all. New scripts/ops/contract-page-audit.py
measures the whole page the way a browser loads it — concurrently,
scored on the SLOWEST panel — because the per-endpoint harness
reported every one of these reads as passing. - SECURITY (live surface): a captured passkey sign-in was an
unlimited, never-expiring session mint.
POST
/v1/auth/passkey/finish-login accepted a replay of the same
ceremony cookie + assertion body indefinitely: the ceremony carried
no server-side expiry, and nothing marked a challenge used. The
expiry was believed to be covered — the guard was written — but
go-webauthn only stamps SessionData.Expires when
Config.Timeouts.<ceremony>.Enforce is true and that field defaults
FALSE, so Expires was always the zero time and the check was dead
code. The only bound was the cookie's Max-Age, which is a browser
hint an attacker's HTTP client ignores. Two fixes: the timeouts are
now configured (5 minutes, enforced) and an unstamped ceremony is
refused rather than treated as eternal; and each challenge is now
SINGLE-USE, spent through a Redis-SETNX guard (passkey:ceremony:*,
the same mechanism the SEP-10 replay guard uses — F-1224) after the
assertion verifies and before any session is minted. The guard fails
CLOSED: if the store is unreachable the sign-in is refused (500)
rather than granted on trust, and email-code sign-in is unaffected.
Redis-less deployments fall back to an in-process spent-set
(single-instance accounting, warned at boot). Note for reviewers of
the old behaviour: the sign-counter clone check was NOT a backstop
here — go-webauthn deliberately exempts counter 0, which is what
Apple/iCloud passkeys report forever. Regression tests drive the
real ceremony end-to-end against a software authenticator, including
a mint-then-replay. - SECURITY (live surface): passkey sign-in never asked for or
required user verification, making passwordless sign-in
possession-only — whoever held the authenticator was the account, no
biometric or PIN involved.
AuthenticatorSelection was unset and
neither begin call passed a user-verification requirement, so the
library's shouldVerifyUser was false, the UV bit was never
checked, and the options JSON omitted the field entirely (browsers
then applied their own default). Both ceremonies now require user
verification. Trade-off, taken deliberately: a security key with no
PIN configured can no longer be enrolled or used as a first factor. - A passkey label with 34+ multi-byte characters 500'd instead of
saving. The name was truncated by BYTES while the storage CHECK
counts CHARACTERS, so a CJK label was cut mid-rune, and Postgres
rejects invalid UTF-8 — after the authenticator had already burned a
resident-credential slot for a credential the server then never
stored. Truncation is now by runes.
- "Body too large" was unreachable on four auth endpoints
(
/v1/auth/login, /v1/auth/verify-code, both passkey finish
routes): io.ReadAll(io.LimitReader(…)) returns a nil error at its
cap, so an oversize body was silently TRUNCATED and then surfaced as
a confusing parse error. All four now use http.MaxBytesReader, the
pattern the rest of the repo already follows. - `/v1/accounts/{g}/positions` runs its six protocol folds in
parallel (sub-second audit's last warm breach, 1.99s): the folds
are independent Postgres reads and were executed serially, so the
endpoint's latency was their sum rather than their max. Output is
byte-identical — each fold writes its own slot and the results plus
coverage notes merge in the original fixed order. Fixing this also
required making the shared per-request asset resolver
concurrency-safe: it memoises into a plain map, and concurrent map
writes are a FATAL runtime throw no recover() catches, so the
parallel folds would have crashed the process under load.
Fixed
- Protocol pages keep their bespoke visual suite when the battery
misses its budget (§2.6b grounding incident): the detail VIEW has
been prewarmed + stale-served since 2026-07-31, but the bespoke block
INSIDE it had no cache of its own — it is built last, so it inherited
whatever was left of the rebuild's 90s budget, and when that ran out
(
protocol bespoke build failed … context deadline exceeded) the
block was dropped and, on a key with no healthy entry yet, the
suite-less view was cached and stamped fresh. The block now has a
last-good cache with a detached, single-flighted, gate-classed
(protocol_bespoke, its own served-tier gate — these are Postgres
queries, not lake scans) refresh: a build serves the previous block
instantly and never blocks, only a true first-ever miss computes
inline (bounded by its caller's context, with the compute surviving
it so the next build lands warm), and a failed or starved refresh
keeps the last good block. A block older than 45 minutes (≈3 prewarm
sweeps) is still served but reported: analytics.status gains a
stale value, distinct from unavailable, and such a build now
counts as COMPLETE for cache displacement instead of being pinned out
as degraded. - `/v1/network/throughput` is prewarmed and snapshot-served: the
/network page's daily series is a FINAL scan over up to a year of
stellar.ledgers with three argMax columns, and it ran inline on the
8s request budget — so a cold or loaded first load lost the panel
(the "no operations in 24h" half of the same incident) and, because
the scan died with the request, no retry could land warm. It now
rides the established SWR shape (5-minute TTL matching the API's
5-minute prewarm loop, detached single-flight refresh under the
network_throughput gate class, stale entries served with
flags.stale + their real as_of). ONE entry holds the maximum
365-day window and every request slices its tail, which also collapses
the key space: an unauthenticated caller walking ?window_days=1..365
previously bought 365 distinct year-class scans. partial is now
decided at serve time, so a cached series that crosses UTC midnight
no longer advertises a complete day as still accumulating.
Fixed
- Explorer: absent data no longer renders as a factual zero
(frontend-honesty sweep, follow-on to the CCTP / roster /
/network
incident in docs/operations/v1-launch-plan.md §2.6b). A whole class
of surfaces coalesced a MISSING value — an expensive aggregate the API
honestly omitted on a budget miss, a 503 from an 8s query ceiling, a
build-time transport blip — into ?? 0 / ?? [], then published the
result as an empirical claim about the chain. Absent now renders —
or an explicit "unavailable" affordance; a served zero is still
rendered as `0` / "no X", which is the entire point of the
distinction. Fixed:
- /dexes/{source} + /exchanges/{name}: a /v1/markets 503 claimed
"No pools/pairs found in the last 14 days" (and "0 on this page").
- /exchanges: the CEX pair board is a Promise.all over four venue
fetches — one 503 headlined "0 CEX pairs · No CEX pairs reporting".
- /dexes, /oracles, /aggregators: a failed /v1/sources read
claimed Stellar has no DEXes / no oracles / no aggregators.
- /issuers/{g}, /issuers long-tail shell, and the issuer panel on
every asset page: /v1/issuers/{g} SOFT-FAILS its per-asset fan-out
(error *or* deadline), so absent assets was baking "Assets 0",
"Total observations 0", "Issued assets (0)" and "No issued assets
observed" for issuers with live assets. Unknown first/last-seen
ledgers also rendered as #0, a ledger that cannot exist.
- /assets/{slug} liquidity tab: a bespoke fetcher swallowed 5xx,
429 and its own timeout into [], baking "No DEX pools observed
touching {code}" into the static export.
- /assets/{slug} supply tab: a failed /v1/chart asserted "No
market-cap history for this asset".
- /external/assets/{slug}: any transport failure baked the flat
denial "We don't track an external asset with the slug X"; only an
authoritative 4xx may say that now.
- /lending/{pool}: an empty listing (what the API serves when no
lending reader is wired) baked "Auctions (total): 0".
- /sources/{name}: a null market read baked "0 pairs · No markets
observed for this source".
- /status: an unreachable latency backend rendered "0.0 ms" in green
(a perfect-SLO claim from a missing measurement) and a failed
freshness probe rendered "0 / 0" active sources.
Each fix ships a render test asserting BOTH directions — absent →
—/unavailable, served zero → 0/"no X".
Fixed
- `/v1/accounts/{g}/transactions` 6.7× faster (sub-second audit
2026-08-13, r1-measured): both union arms carried the WIDE tx column
set (memo, result_code, source_account, …) through their own scan and
sort of
stellar.transactions, and the outer DISTINCT then
materialised both. The query now resolves the KEYSET in the union and
hydrates the wide columns once over the surviving ≤limit keys —
1.479s → 0.219s for the same 50 rows, with the cross-arm dedupe now
provided by the hydration pass's LIMIT 1 BY. - `/v1/accounts/{g}/operations` 2.7× faster — same two-phase shape,
and it matters more here:
opCols carries body_xdr, the column the
code itself measures at ~600ms over the 24B-row table, and both arms
were paying it. 0.407s → 0.153s (r1, 50 rows).
- CI integration gate stopped failing on the clock: the suite's
go-test deadline is raised 20m→35m. It hit the ceiling on three
consecutive pushes with the running test 1s in, while the same suite
completes in ~13m locally (CI runners are ~1.5× slower) — a gate that
reports "the clock ran out" as a failure stops being a signal. The
next raise should split the suite by package instead.
- `POST /v1/register` returned keys that could not authenticate
(found in the v0.32.0 post-deploy battery): the mint wrote only the
Postgres MANAGEMENT row, but r1's auth middleware validates against
the REDIS store (
backend=redis), so a freshly registered key 401'd
on first use — a 200 response carrying a dead credential, worse than
an honest failure. The mint now mirrors the same plaintext into the
validator's own store (RedisAPIKeyStore.CreateWithSecret) whenever
that store is wired, and a mirror failure fails the request instead
of handing back a key that cannot work. The agent-onboarding flow is
functional again.
Added
- Passkey (WebAuthn) sign-in for the dashboard: six new
endpoints under
/v1/auth/passkey/ — begin-login /
finish-login (anonymous, usernameless discoverable-credential
flow; finish mints the SAME session cookie the email-code flow
does, via the shared session-mint path), begin-register /
finish-register (session-gated; resident key required so the
credential can sign in usernameless), and credentials
(GET list + DELETE {id}, session-gated, owner-scoped). Server
is github.com/go-webauthn/webauthn v0.17.4; RP ID/origin derive
from the existing api.dashboard.base_url. Ceremony state rides
a 5-minute HMAC-signed HttpOnly cookie (purpose-bound so a
registration challenge can't finish a login); a sign-count
regression (possible cloned authenticator) refuses the login and
logs. Storage is the new webauthn_credentials table (migration
0140, additive). Explorer: "Sign in with a passkey" on /signin
(feature-detected) + a Passkeys list/add/remove card on
/dashboard/settings. OpenAPI paths + all three generated
artifacts refreshed; SDK triage recorded in
uncoveredOperations.
Security
- Dashboard 6-digit sign-in codes are no longer derivable from
the database (parked audit finding, aggregate+dashboardauth
cold audit 2026-08-03): the code was an unkeyed public function
of
magic_link_tokens.token_hash (base32 of its first 4 bytes),
so any Postgres read — SQL injection elsewhere, a stolen backup —
yielded every in-flight sign-in code directly, no brute force
needed, and with it a session for any address the reader could
trigger a login for. The code is now
HMAC-SHA256(server_secret, token_hash) reduced to 6 digits —
same UX, same storage, one derivation swapped; the secret lives
in config/env (api.dashboard.code_secret_env, default
STELLARINDEX_DASHBOARD_CODE_SECRET), never in Postgres. With
the env unset the API falls back to a random per-process secret
(still keyed; in-flight codes just don't survive a restart —
they live 15 minutes and the magic link is unaffected). Deploy
note: codes emailed before the deploy stop verifying for their
remaining TTL; links keep working.
onboarding path**: one unauthenticated POST (empty body fine;
optional name + contact-only email, never verified) creates a
free-tier platform account and mints its first Postgres-backed API
key, returning {account_id, api_key, key_id, key_prefix, tier,
limits} with the plaintext shown once. Rides the same per-IP
signup throttle as /v1/signup (shared budget → 429) and the
signup Content-Type CSRF gate. OpenAPI path + all three generated
artifacts refreshed (docs-api, docs-postman, web-generate-api);
SDK triage recorded in uncoveredOperations; agent-facing
walkthrough at docs/agent-onboarding.md.
Changed
- Tier model collapsed to `anon` / `free` / `partner` (follow-up
to the Stripe removal — the platform is free).
free is every
registered account's default, anchored to the old Starter numbers
(1000 req/min, 1M req/month, 25 keys, 10 webhooks, 25 price
alerts); partner is staff-set per-account limits via the existing
PATCH /v1/admin/accounts/{id} override + key-clamp path, with the
old Enterprise numbers as ceilings when no override is set
(100k req/min, 1B req/month, 250 keys, 100 webhooks, 1000 alerts);
anon documents the unauthenticated 60/min per-IP baseline.
Legacy stored tier strings map in code
(platform.Tier.Canonical: starter→free,
pro/business/enterprise→partner; unknown fails closed to free) and
writes fold back to CHECK-legal strings
(platform.Tier.StorageValue) — migrations untouched. The admin
PATCH accepts both vocabularies and canonicalises.
Removed
- Stripe/billing integration removed — the platform is free
(operator decision 2026-08-10: anonymous access, free accounts,
staff-set partner limits; no payments). Deleted the
POST /v1/webhooks/stripe endpoint (handler + route + OpenAPI path
+ generated artifacts), [api.stripe] config
(STELLARINDEX_STRIPE_WEBHOOK_SECRET), platform.BillingStore /
Subscription / StripeEvent and their Postgres store,
Account.StripeCustomerID + GetByStripeCustomerID, the
stellarindex_stripe_platform_sync_errors_total +
stellarindex_stripe_dead_letters_open metrics with both alert-rule
trees and their runbooks, and paid-plan copy in the explorer
(pricing/signup/company/dashboard now describe free access).
The shared tier-clamp machinery the admin
PATCH /v1/admin/accounts/{id} path uses survives in
internal/api/v1/keybudgets.go (StripeKeyManager →
SelfServiceKeyManager). Migrations are untouched — historical
stripe_* columns/tables stay in place, unused.
Fixed
- Contract WASM view resolves pre-capture contracts ("this
contract's on-chain WASM isn't in the captured ledger window yet",
operator report 2026-08-11): the instance→hash hop now reads the
genesis-complete
contract_instance_changes index first, so any
contract whose instance was ever written resolves its current
executable (or SAC verdict) regardless of the live-capture window;
the code-bytes hop was already lake-complete (r1-measured: all
4,534 contract_code keys present). Legacy read remains the fallback. - Trade USD valuation: divergent-leg cross-check (fake-XMR
incident 2026-08-11): an attacker planted an INDUSX/XLM bridge rate
for the cost of the $0.01 dust floor and two no-XLM-leg trades were
stamped ~$91M each off the poisoned quote-side rate (real value
<$0.01 — a $182M fake spike in the SDEX volume series). The FX tier
now values BOTH legs through the resolver when possible and stores
the SMALLER when they disagree beyond 10× — inflating a print now
requires pumping both legs' markets with real value. The two
poisoned rows re-derive to honest values via the generation-guarded
corrective path.
Fixed
- Detached-refresh gate is now class-fair (inventory #26 item 5,
second half): the single global bound stopped the unauthenticated
scan-amplification but let one key class starve the rest — a crawler
churning fabricated contract ids could hold every slot while cold
account/holders/directory pages fast-503d behind it. Each refresh
class (account state, contract detail, asset holders, contracts
directory) is now additionally capped at half the global limit; the
global pool-safety bound is unchanged.
- /v1/contracts directory census: 40s scans replaced by a day-keyed
rollup (inventory #26 item 2 — the single heaviest explorer read,
~160 runs per 3h across the prewarm rungs). New
stellar.contracts_census_daily (plain per-day per-contract counts;
whole days recomputed and swapped via REPLACE PARTITION — no MV, so
the Summing double-count class cannot arise) maintained by the new
stellarindex-ops ch-census-rollup on a 30-min timer; the reader
sums day rows (sub-second) with a coverage check that falls back to
the exact scan while a backfill is incomplete. Window floors round
to UTC-day resolution. - Aquarius `claim_protocol_fee` now records WHICH token was claimed
(sources-decode audit 2026-08-04, finding 5): the token address lives
in
topic[1] — not the body — and migration 0129 shipped no token
column on the documented premise that a recent trade could resolve
it; the lake refutes that (one tx claims two different tokens with
near-identical amounts), and per-pool SUM(amount) without the token
adds integers of different token scales. FeeEvent gains Token
(decode refuses a claim without it), migration 0139 adds the nullable
column, and the 163 token-less rows already on r1 re-derive via
projector-replay -source aquarius (queued). - DeFindex `harvest` events are now decoded (sources-decode audit
2026-08-04, finding 4): the recognise-and-drop premise ("body never
observed on-chain") was disproved by the lake — 1,018 harvests with
body
{amount, from, price_per_share}, the exact shape decodeFlow
reads by name. Harvests now emit direction='harvest' strategy-flow
rows (migration 0138 widens the CHECK; user-position sums exclude
them by construction — harvest is strategy yield, not a user flow).
Historical recovery via projector-replay -source defindex (queued). - Phoenix pre-upgrade swaps no longer dropped (sources-decode
audit 2026-08-04, finding 1 — HIGH): the pre-upgrade pool WASM
(ledgers 51,019,036 → 53,134,167) emitted 7 field-events per swap —
no "actual received amount" — but
RawSwap.Complete() required that
slot even though decodeSwap deliberately never reads it, so all
5,161 pre-upgrade swaps aged out as orphans (r1-confirmed: zero
phoenix trades before ledger 53,134,242). Aged-out groups whose
decode-consumed slots are present are now decoded at sweep time
instead of orphaned; the current era's eager 8-field emit and orphan
accounting are unchanged. Recovery of the historical rows needs
projector-replay -source phoenix -from 51019036 (queued). - `/v1/contracts/{id}/code-history` cold reads (the last persistent
503 class in the route sweep): new keyed
stellar.contract_instance_changes index — an MV-fed
ReplacingMergeTree holding one narrow row per captured
instance-entry write with the executable verdict pre-extracted via
fixed-offset XDR substrings (byte-verified against go-stellar-sdk
marshalling and against live r1 data). The reader walks the
contract's primary key instead of a scan-shaped key_xdr predicate
over the whole changes log; legacy scan remains the fallback where
the index is absent. Historical fill via the new
stellarindex-ops ch-instance-backfill (windowed, resumable, under
run-heavy-job.sh).
Fixed
- /ledgers table was permanently stuck on "Loading…" — the page
wrapped
LedgersTable (which takes no useSearchParams) in a
vestigial <Suspense fallback={null}>, and the static exporter
emitted that boundary as a never-completing pending template, so
browsers never hydrated or client-rendered the subtree (zero network
activity; the only such boundary on the site — audited all pages).
Wrapper removed. The live-follow refetch on /ledgers + /operations
is also throttled to one per 10s (operations' newest row structurally
trails the ingest-tip stream, so unthrottled it refetched every
close, ~12 req/min per viewer) and no longer fires while the initial
page fetch is in flight.
Fixed
- `/v1/price/stream` no longer interleaves aggregation windows on
one topic (cold audit 2026-08-03, r1-confirmed: three consecutive
price_update events carried window_seconds 300/3600/86400 with
materially different prices, so a client reading value_decimal saw
the price flap three times per tick). The Hub topic key now includes
the window (closed:<asset>/<quote>/<window_seconds>) and the
stream accepts ?window_seconds= (default 300) to pick one series.
Also fixed: subscribing with an alias spelling (?asset=native vs
the aggregator's crypto:XLM) silently matched nothing forever —
the handler now subscribes to every alias spelling of the pair. - `/v1/observations` (+ its stream) now scans every alias spelling
of the pair — CEX observations live under
crypto:XLM while SDEX
legs live under native, so a ?asset=native query was silently
blind to the CEX rows (and vice versa). Alias results merge keeping
the newest trade per source; single-spelling pairs still do one scan. - `/v1/price/stream` events now carry the documented `/v1/price`
envelope shape from BOTH producers (
data + as_of; flags /
sources only when evaluated). Previously the aggregator bridge
emitted {asset, quote, window_seconds, value_decimal, observed_at}
and streampublish emitted {snapshot, sources, stale} — two bespoke
shapes on one endpoint, neither matching the OpenAPI example. No
fabricated flags: an absent flags object means "not evaluated",
never "fresh". as_of is the bucket end, keeping cross-region
payloads byte-identical (ADR-0015).
- `/v1/price/tip/stream` now shares one tip-compute loop per
distinct (asset, quote, window) across all connections via the
streaming Hub, instead of running a private 5-second query loop per
connection (cold audit 2026-08-04: "tip stream = 6 DB queries/s PER
CONNECTION — pool saturates at ~2300 streams"). Steady-state DB cost
now scales with distinct pairs being watched, not with viewer count;
producers linger 30 s after their last subscriber leaves to absorb
reconnects, and Hub resume (
Last-Event-ID) now works on the tip
stream. Per-connection pre-flight verdicts (404 / withheld / 400)
and the instant first frame are unchanged. Hub-less deployments keep
the legacy per-connection producer. This is the scaling precondition
for the explorer's live-ticking pages (RT-1). - The accounts hub's most-held chart no longer includes native XLM —
every funded account holds XLM by definition, so charting it
collapsed the issued-asset bars to slivers.
- CI's web advisory gate: pnpm override floors raised for four fresh
high GHSAs (nanoid, undici, brace-expansion, js-yaml — the prior
js-yaml range
>=4.2.0 <4.3.0 excluded the patched 4.3.1).
Added
- Explorer live ticks (RT-2): a shared SSE multiplexer
(
web/explorer/src/lib/live/) — one refcounted EventSource per
endpoint per tab, slow reopen on hard failure (WB-04) — now powers a
live network heartbeat in the sidebar (latest closed ledger, pulsing,
linking to its /ledgers page) and live tip-price streaming on asset
pages: the headline price ticks in real time with an up/down flash
and a "live tip price · streaming" caption, degrading to the
existing 60s poll + baked value when the stream is unavailable.
The home strip's XLM cell and the market-pair headline (previously
a BUILD-frozen price captioned "as of <build time>") stream the
same way, and /ledgers + /operations follow the network live: new
rows animate in on every ledger close while page 1 is on screen
(paging into history pauses following). The markets board follows
ledger closes with a throttled refresh and per-cell price flashes —
one SSE connection for the whole table. Animations respect
prefers-reduced-motion. - Contract activity card (insight program unit 1): every contract
page shows first-seen / last-seen ledgers, lifetime active-ledger
count, and a 30-day activity sparkline — µs reads off the
contract-keyed
contract_active_ledgers index.
Added
- Accounts hub analytics (operator request 2026-08-08; API 1.19.0+
/v1/accounts/stats): network totals (funded accounts, trustlines,
XLM held), balance statistics (avg/median/p90/p99 — stroops as
strings, ADR-0003), top-100 concentration, a log10 wealth-
distribution histogram, trustlines-per-account bands, and the
most-held assets board — all computed by the same 30-minute
ch-holders-rollup cycle (the analytics ride the scans the holders
boards already pay for) and served from keyed tables, sub-second by
construction. The /accounts page grows a stat strip, two
distribution charts, and the most-held board above the wealth
directory.
Showing the 40 most recent of 217 releases. Full changelog →