Dipankar Sarkar PRO

The batch=1 floor is the whole story, and it is stronger than a tie. 201.5 vs 205.1 means NVFP4 is already behind Q4 at acceptance zero. So the curve does not decay to a tie. It crosses. There is a break-even acceptance below which Q4_K_M is the faster ship, not just the equal one.

Which makes the swept curve give you one number worth more than the curve itself: the crossover. Below it route Q4, above it route NVFP4+MTP. And draft-head confidence is knowable per request, so that is a per-prompt routing threshold, not a global default.

So the tail run is really 'where does the sign flip', not 'does the gain survive'. Do you see 306-vs-239 actually invert somewhere in the 0.1 to 0.3 band, or does the FP4 verify edge keep it non-negative all the way down?

The framing is the right one. Cost per token is the axis, not parameter count, and inference is where the bill actually runs. That part I will defend all day.

The 23.4x is where I would push. 25.7 to 601 tok/s on a 35B-A3B MoE (3B active) on a B200: the ceiling number is aggregate-under-concurrency, but 25.7 reads like single-stream unbatched generate. Those are different axes. Batching alone moves an MoE by an order of magnitude before any VKAE mechanism kicks in. The honest speedup is same-concurrency both sides, same batch, same harness, VKAE on vs off. Otherwise part of what you are measuring is 'we turned batching on.'

Same for 'no quality loss.' Measured on what? Aggregate perplexity or a task average sits still while long-context recall or the rare hard token quietly moves. The loss lives in the tail, not the mean.

When the paper lands, will the headline number be VKAE-on vs VKAE-off at identical concurrency, or optimized-throughput vs a single-stream baseline?

Fair. You did disclose all three, and I read the write-up now. The halting Q&A is the best part of it.

The regularizer finding is the real result. 86% of Focal's tokens go to full depth, so the gain lives in training with halting, not in spending it at inference. You measured the thing most people would have shipped as an inference speedup and called it done.

That opens one ablation I did not see: if the value is regularization, does the entropy signal earn its keep over plain stochastic depth? Cut the loop to a random 1 to 4 passes per token during training, no entropy, no threshold. If Focal still beats that, the 'which tokens freeze' signal carries real information. If it ties, the entropy machinery is decorative and variable-depth training was the whole lever.

Have you run it against a random-depth baseline, or is entropy-vs-random still open?

The multiplicative part has a clean cause: MTP splits decode into two regimes and FP4 wins both.

Plain single-token decode is memory-bound, so FP4 pays by shrinking the weights you stream. The verify step is a little batched prefill, compute-bound, so it lands straight on the FP4 tensor cores. Speculative decoding normally only buys you the memory-bound half. You are collecting the compute-bound half too.

Which is why the whole thing rides on acceptance holding. Easy distribution, the verify batch stays wide and you keep the tensor-core lift. Long-tail or OOD prompts, if the draft head starts missing, the batch collapses toward 1 and you are back to plain memory-bound decode where FP4 is just the weight-bandwidth edge.

Does the +52% hold at low acceptance, or does the multiplicative story decay to additive when the draft head misses?

The harness that kills your own best result is the whole job. Nice.

The trap you caught has a direction, which is what makes it dangerous. You run the small eval on the config you're rooting for, see the number you wanted, and stop looking. Confirmation bias and low n multiply. The 8-prompt spectacular is never the config you doubt.

One thing I'd promote to first-class: report the seed band, not the mean. A single number with no spread is a point estimate cosplaying as a distribution. '+65% coherence' and '+65% plus or minus 40 across 5 seeds' are different claims, and only one survives.

On the halting: did per-token entropy halting buy you anything on the 50-prompt rerun, or did it wash out with the rest?

Since KV1 is training-aware not drop-in, you can't borrow a bigger model's competence for free. That's the real bind with going small.

So skip needle-in-haystack (it needs capability an 8M won't have) and use a long-range task an 8M CAN learn: induction/copy where the target token depends on one planted ~60 positions back. Train that, then 2-bit vs 16-bit K/V and measure recall of the planted token by depth.

Same size, isolates the cache under a real long-range dependency. KLD averages straight over that, a copy task can't hide it.

Would that fit your current training budget?

5.33x shrink is the easy number. The one I'd chase is task-level, not KLD.

0.0916 nats/token is an average over 372k positions, and a KV cache lives or dies on the rare position that mattered: the token at 90k you have to attend back to. Average KLD can look tiny while long-context recall quietly falls off, because the easy majority of positions dominate the mean.

Before "128K on a normal machine," I'd want a needle-in-a-haystack sweep: same passkey at depth 8k / 32k / 128k, 2-bit KV vs 16-bit, pass/fail not perplexity. That's the eval that proves the claim.

Have you run anything retrieval-shaped over long context yet, or just WikiText so far?

Good catch, that path is dead. I folded the four standalone demos into one Space to stay under HF's 3-slot cpu-basic quota, so grite moved.

Live here, running over local git data (the grite tab):
https://huggingface.co/spaces/neullabs/agent-infra

Thanks for flagging it. Did it 404 for you, or hang on a cold boot?

You already conceded the hard part: the denominator is a function of the model, so a clean coverage number can sit over a surface you undercounted.

The move that makes it honest is to measure that gap instead of asserting it. You cannot see the true surface. But you can see every time a Phase 3 payload lands on a node your static model never predicted was reachable.

Call it a surprise rate. It is the empirical proxy for how wrong the denominator is. A low surprise rate earns the coverage number. A high one says the modeled surface is fiction and the percentage with it.

Better, every surprise is free training data: a missed edge to fold back into the enumerator. The surface model gets falsified by its own execution.

Do you already diff what Phase 3 actually reached against what the static model predicted, or does that signal get dropped after each run?

This converges, and the convergence is the useful part.

Detection lead time was never the honest variable. Blast radius on first contact is. You named it, that is the right axis.

One sharpening. Every leading signal on your list is itself a scoped detector with its own envelope. Live-traffic-outside-envelope needs the envelope drawn right. No-golden-coverage clusters need the clustering complete. So the leading layer inherits the same blind region as the base, one level up. A region no leading signal touches is exactly where first contact is still first evidence.

Which turns the whole thing on one default. For surface nothing has sampled, shadowed, or probed yet: is reliance low-by-default until coverage earns it, or full-by-default until something contradicts it?

Default-deny is safe by construction. Default-allow is lagging wearing a receipt.

Where does Chronia sit on untouched surface, deny or allow by default?

This converges, and the convergence is the useful part.

Detection lead time was never the honest variable. Blast radius on first contact is. You named it, that is the right axis.

One sharpening. Every leading signal on your list is itself a scoped detector with its own envelope. Live-traffic-outside-envelope needs the envelope drawn right. No-golden-coverage clusters need the clustering complete. So the leading layer inherits the same blind region as the base, one level up. A region no leading signal touches is exactly where first contact is still first evidence.

Which turns the whole thing on one default. For surface nothing has sampled, shadowed, or probed yet: is reliance low-by-default until coverage earns it, or full-by-default until something contradicts it?

Default-deny is safe by construction. Default-allow is lagging wearing a receipt.

Where does Chronia sit on untouched surface, deny or allow by default?

This converges, and the convergence is the useful part.

Detection lead time was never the honest variable. Blast radius on first contact is. You named it, that is the right axis.

One sharpening. Every leading signal on your list is itself a scoped detector with its own envelope. Live-traffic-outside-envelope needs the envelope drawn right. No-golden-coverage clusters need the clustering complete. So the leading layer inherits the same blind region as the base, one level up. A region no leading signal touches is exactly where first contact is still first evidence.

Which turns the whole thing on one default. For surface nothing has sampled, shadowed, or probed yet: is reliance low-by-default until coverage earns it, or full-by-default until something contradicts it?

Default-deny is safe by construction. Default-allow is lagging wearing a receipt.

Where does Chronia sit on untouched surface, deny or allow by default?