What does block propagation
latency actually cost a validator?

The identification

A deadline nobody chose#

The consensus spec tells an attester to vote 4,000 ms into the slot, for whatever head it can see at that instant. A block landing at 3,950 ms is votable. At 4,050 ms it is not: the attester votes for the previous block and forfeits TIMELY_HEAD. Nobody chose that cutoff, and no proposer can steer gossip to land on one side of it with millisecond precision. That is a textbook sharp regression discontinuity. RANDAO assigns attesters to slots, so exposure to a late block is as-good-as-random.

I am not comparing fast operators to slow ones; that comparison is hopelessly confounded. I compare the same validators across slots that, by lottery, happened to have an early or a late block.

The three channels

Latency doesn’t cost you once. It costs you three ways.#

The three channels have completely different shapes. Attestation losses are tiny but constant: a slice of one validator's pay, shaved off every time a block runs late. Proposal losses are the mirror image: rare, but when a slow block gets reorged out the proposer loses everything at once. And MEV isn't a loss at all: it is extra income a proposer could capture by waiting longer to publish, if faster propagation bought back the safety margin that waiting spends.

Rolling all three into one "cost of latency" number would hide the thing a buyer actually needs to know: faster propagation barely moves the first two, and almost all of its dollar value sits in the third, which only pays if the operator changes how it behaves. That is why every step below prices the channels separately.

(There is also a fourth channel, D: bandwidth, priced in section 07. It is kept apart because it cuts server costs rather than earning validator revenue.)

Xatu is ethPandaOps’ public beacon-chain dataset: plain parquet over HTTPS, no auth. DuckDB reads it in place and prunes columns, so a query touching 3 of 40 columns downloads 3 column chunks. No warehouse, no ETL. The unit is the slot.

from src.xatu  import connect, XatuPaths, ATTESTATION_DEADLINE_MS   # 4000 ms
from src.panel import build_slot_panel

build_slot_panel(con, XatuPaths(cfg.xatu_base_url), dt.date(2025, 6, 11),
                 min_sentries=cfg.min_sentries, local_dir=cfg.raw_dir)
df = con.execute("SELECT * FROM slot_panel ORDER BY slot").df()

# Treatment: MEDIAN arrival across >=5 sentries, joined on the CANONICAL block root,
# so a reorged competitor block cannot pollute the timing.
df["late"] = (df.arrival_ms > ATTESTATION_DEADLINE_MS).astype(float)

Four things in that SQL are load-bearing: the treatment is the median arrival across ≥5 sentries joined on the canonical block root (a reorged competitor must not pollute the timing); attestations are read from day D and D+1 (an attestation may land 32 slots later; truncation would masquerade as a missed attestation); attesters are de-duplicated at the (slot, committee, voted-root) grain with list ops, never UNNEST (which explodes one day into ~200M rows); and the panel left-joins from proposer_duty, so a slot with no block survives as a missed proposal instead of vanishing.

And a second one, found while building this notebook

A missed proposal has no block, so it has no arrival_ms, and no blob_count, tx_count or payload_bytes either. The covariates vanish exactly when the outcome fires. Any model of the Proposals channel built on own-block features is silently fitted to a sample from which every event has been deleted. Measured on this panel:

Ninety-nine percent of the rows survive, and 100% of the events die. A model fitted there would report a beautiful R² on a question it can no longer see. The fix is lagged network conditions, which come from a slot’s neighbours and therefore exist for every duty, block or no block:

# A missed proposal has NO block -- so no arrival_ms, and no blob/tx/payload either.
# The covariates vanish exactly when the outcome fires. So the Proposals channel must be
# modelled on LAGGED conditions, which come from a slot's NEIGHBOURS and exist for every duty.
g = df.arrival_ms
df["prev_arrival"]  = g.shift(1).ffill()
df["roll_arr_mean"] = g.shift(1).rolling(32, min_periods=4).mean().ffill()
df["roll_arr_p90"]  = g.shift(1).rolling(64, min_periods=8).quantile(.90).ffill()
df["roll_late"]     = df.late.shift(1).rolling(64, min_periods=8).mean().ffill()

This repo was originally scoped as a staggered DiD on mump2p adoption by the seven publicly-named Optimum partners. That study cannot be run, and the reasoning is worth more than the study would have been.

1 · There is no staggering. All seven were named in a single press release on 2025-06-24. Callaway–Sant’Anna exists to exploit adoption-timing variation; with one cohort it degenerates to a 2×2 DiD.
2 · The treatment never happened in the data. mump2p has never run on Ethereum mainnet: it is a Hoodi/private-testnet product, and Xatu is mainnet. The treated operators’ mainnet validators were never treated: the treatment indicator is identically zero across the panel.

You cannot estimate the effect of a treatment that did not occur. So the honest thing to put in a notebook is not a fabricated effect. It is a null calibration: run the full machinery against a placebo where the truth is τ = 0 and confirm it finds nothing. An estimator that finds effects is worthless unless it also fails to find them when they are absent.

import statsmodels.formula.api as smf
from linearmodels.panel import PanelOLS

# mump2p never shipped to MAINNET, so on Xatu the treatment indicator is IDENTICALLY ZERO.
# You cannot estimate the effect of a treatment that did not occur. What you CAN do is prove
# the estimator does not hallucinate one. Placebo: the truth is tau = 0.
df["treated"] = (df.proposer_index % 14 < 7).astype(int)     # 7 pseudo-"partners"
df["post"]    = (df.slot > df.slot.median()).astype(int)
df["did"]     = df.treated * df.post

m = smf.ols("correct_head_rate ~ treated + post + did", data=df).fit(
        cov_type="cluster", cov_kwds={"groups": df.epoch})

fe = PanelOLS.from_formula(
        "correct_head_rate ~ did + EntityEffects + TimeEffects",
        data=df.set_index(["operator", "epoch"])
     ).fit(cov_type="clustered", cluster_entity=True)

Hoodi · the one network where mump2p runs

Why the DiD didn’t just move to the testnet#

The obvious rejoinder to step 02: mump2p is deployed somewhere: Ethereum’s Hoodi testnet, since 2025-06-24. So I went there. Two things break the textbook design.

First, timing. Xatu’s attestation coverage for Hoodi only begins in late October 2025, after the deployment window this study needs: the partners were announced 2025-06-24 and the vendor’s results were captured in September–October 2025. Attestation outcomes (head votes, inclusion distance) are available from November 2025 onward and can be studied today, but a difference-in-differences needs outcomes on both sides of the adoption date, and the pre-adoption side was never recorded. For the window that matters, identification has to run on propagation physics instead.

Second, and more fundamental: there is still no staggering. The operator clusters all move on the same dates, so “adoption timing” never becomes a usable second difference, with or without attestation data.

Why mainnet Xatu carries the study anyway. The money is on mainnet: real MEV, real rewards, the real validator set, none of which exist on a testnet, and Channel C is ~90% of the dollars. Hoodi answers a different and narrower question (“did the overlay visibly carry blocks?” Answer: no, as far as public data can see), and it answers it with only 4 sentries feeding the libp2p layer. So: mainnet for the physics and the pricing, Hoodi for the adoption forensics, and the adoption-detection machinery built for that search is reusable on mainnet the day a real rollout happens. That is the staggered DiD, pre-built and waiting for its treatment.

The channels that matter are rare-event channels: 0.68% of duties miss a proposal; 0.83% of blocks cross the deadline. Rare events mean wide intervals, and wide intervals mean you cannot price the product. CUPAC buys precision without buying more data: predict the outcome from pre-treatment covariates, then use the out-of-fold prediction as a control variate. Variance falls by roughly corr(Y, Ŷ)².

import lightgbm as lgb
from sklearn.model_selection import TimeSeriesSplit

# CUPAC: predict Y from pre-treatment covariates; use the OUT-OF-FOLD prediction as a control
# variate. Var falls ~ corr(Y, Y_hat)^2. Out-of-fold is not optional -- predictions fitted on
# the rows they adjust would absorb the treatment effect itself. Folds must be TIME-ORDERED.
oof = np.full(len(f), np.nan)
for tr, te in TimeSeriesSplit(n_splits=5).split(X):
    g = lgb.LGBMRegressor(n_estimators=400, learning_rate=.05, num_leaves=31,
                          min_child_samples=40, random_state=SEED)
    g.fit(X[tr], y[tr]);  oof[te] = g.predict(X[te])

base  = smf.ols("_y ~ _d",         data=f).fit(cov_type="cluster", cov_kwds={"groups": f.epoch})
cupac = smf.ols("_y ~ _d + cupac", data=f).fit(cov_type="cluster", cov_kwds={"groups": f.epoch})

First: prove the implementation works

Before trusting CUPAC on real data, I check that the implementation can shrink an interval when there is genuinely signal to exploit. The test is a synthetic dataset where the answer is known: the outcome is built partly from the covariates (so a model really can predict some of it), a randomised treatment with a true effect of τ = -0.25 is added, and the rest is noise. Theory predicts the standard error should fall by 1 − √(1 − R²).

The validation lands where theory says it should: at R² = 0.250, 1 − √(1 − R²) predicts a 13.4% SE reduction; I measured 14.2%. The estimator is sound. It simply has nothing to work with on the real panel.

Mean latency is the wrong KPI. Mean arrival is 2.35 s, comfortably safe. Yet 0.83% of blocks still cross 4,000 ms, and only those cost anything. A product that shaves 200 ms off the mean and nothing off the p99 is worth approximately zero. So model the arrival distribution.

import pymc as pm, arviz as az

with pm.Model():
    mu    = pm.Normal("mu", 2.3, 1.0)            # grand mean arrival (seconds)
    tau   = pm.HalfNormal("tau", 0.5)            # between-hour sd
    z     = pm.Normal("z", 0, 1, shape=n_hours)  # NON-CENTRED: the centred form funnels
    theta = pm.Deterministic("theta", mu + tau * z)
    sigma = pm.HalfNormal("sigma", 1.0)
    nu    = pm.Gamma("nu", 2.0, 0.1)             # HEAVY TAIL: gossip arrival is not Gaussian
    pm.StudentT("obs", nu=nu, mu=theta[hour_idx], sigma=sigma, observed=y)

    idata = pm.sample(1000, tune=1000, chains=2, target_accept=.9)
    ppc   = pm.sample_posterior_predictive(idata)

p99 = np.percentile(ppc.posterior_predictive["obs"], 99, axis=-1) * 1000   # decision variable

Non-centred because the centred parameterisation funnels and the sampler jams against small τ. Student-t because gossip arrival has fat tails, and a Gaussian would be dragged by them and under-predict the p99, understating exactly the risk being priced. Sampling was clean: max R̂ = 1.000, 0 divergences.

Slots are a time series. A shuffled KFold trains on slot 9,000 to predict slot 100, leaking the future into the past. And because the CUPAC covariate in Step 03 is built from those out-of-fold predictions, the leak would not merely inflate an R²; it would silently bias the very treatment effect it exists to de-noise.

from sklearn.model_selection import TimeSeriesSplit, KFold

# Slots are a TIME SERIES. A shuffled KFold trains on slot 9,000 to predict slot 100 -- leaking
# the future into the past. And because the CUPAC covariate is BUILT from these out-of-fold
# predictions, the leak would not merely inflate an R2; it would silently bias the very
# treatment effect it exists to de-noise.
oof_r2(TimeSeriesSplit(n_splits=5))                      # honest
oof_r2(KFold(n_splits=5, shuffle=True, random_state=0))  # leaks -- and "looks" better

The leaky splitter reports the better number. Both are negative here, which is its own answer, but note the direction: shuffling buys 0.032 of free R² out of nothing but leakage. That is the danger in miniature: the metric improves, the science degrades, and nothing in the traceback tells you.

Measured from 212,162 slots of relay bid traces: the best bid available to a proposer roughly doubles across the slot, then plateaus right after ~3.5 s: builders stop bidding on a block that cannot beat the deadline, because such a block is worthless.

So the commercial argument inverts. A faster transport does not earn MEV by being fast. It earns MEV by buying delay budget: if transit is Δ ms quicker, the proposer can publish Δ ms later and still land at the same arrival time: identical reorg risk, strictly better bid. That conclusion is visible only because the three channels were kept apart.

How that becomes the dollar figure, step by step

1. The slope. dV/dt = 9.12×10⁻⁶ ETH/ms (≈0.0091 ETH per second of delay): the slope of the curve charted above, fit over its 1,000–4,000 ms window (where timing-game proposers actually decide), from the full 30-day June 2025 panel of relay bid traces (212,162 slots). Every dollar figure that follows traces back to this one measurement.
2. The compressible milliseconds. Only transit compresses: the spread between the first and the 90th-percentile sentry sighting of the same block, mean 1,002 ms on the panel. The ~1.4 s proposers deliberately wait to accrue bids is untouchable by any transport. At the vendor's 6×: 1,002 × (1 − 1/6) ≈ 835 ms saved, capped per slot at the headroom left before the 4,000 ms deadline (delay you don't have can't be spent).
3. The behavioural step. The proposer publishes those 835 ms later, lands at the same arrival time, and harvests 835 × 9.12×10⁻⁶ ≈ 0.0076 ETH ≈ $13.75 of extra bid per block. This step is a config change, not an installation; skip it and Channel C pays ~nothing.
4. The frequency. Proposer selection is stake-proportional: 2,628,000 slots/yr ÷ ~881k active validators ≈ 3.0 blocks per validator-year → ≈ $41.0/validator/yr uncapped; applying the per-slot headroom cap on the observed slot mix trims it to $38.13.
5. The fleet. × 147,737 validators across the seven partners = ≈ $5.63M/yr: the Channel C column in the table below, and ~92% of the headline number at the top of this page.

The same model, run for each of the seven named partners at the vendor's headline 6× speedup, plus a fourth channel, D: bandwidth, which is different in kind: it is an opex saving, not revenue. RLNC-coded pub/sub replaces gossipsub's redundant flooding (~4× duplication measured in the literature, ~1.2× residual for the coded scheme), and it scales per node, not per validator. Its dollar value depends almost entirely on where you host: at cloud egress prices ($0.09/GB) it is real money; on a bare-metal box with bundled transit ($0.001/GB) it rounds to nothing. I price both and refuse to pick one.

* nodes ≈ validators / 500 keys per beacon node. Channel D assumes gossip duplication 4.0× → 1.2× under RLNC on ~542 KB/slot of measured wire payload (block + 3.81 blobs), and should be re-measured after PeerDAS ships. TOTAL = A + max(B, C) + D_cloud; B and C are mutually exclusive uses of the same saved milliseconds, D is additive because it spends none of them.

Channel C is not new money; it is repriced auction share. An adopter wins timing-game value partly from proposers who didn't adopt. Sweep the adoption share and the structure falls out: per-adopter value peaks at 40% adoption ($26.04/validator/yr), total captured value peaks at 55% (~$9.8M/yr), and at 100% adoption the MEV edge is exactly zero: everyone waits longer, nobody gains relative position, and all that survives is the non-rivalrous A + B + D (~$3.34/validator/yr).

Two commercial readings, both honest: the seller's: early adopters capture the most per seat, and the spread between adopting and not ($40.61/validator/yr at the plateau) is the real product, since past ~40% you buy it to avoid losing rather than to win; and the buyer's: the pitch deck number assumes your competitors haven't bought it yet.

The JD, answered

Every “experience with X?” → “here’s where I used it on your problem”#