Projects built

How often does an agent reach the right answer through a wrong process?

0.496 vs 0.000

An agent that never follows the correct process scores 0.496 on outcome and 0.000 on trajectory. Its mirror image scores 33.1 points lower on outcome than trajectory — the two metrics disagree in opposite directions.

• python
• pytest

Does agentic multi-step retrieval beat single-shot RAG, and what does the loop cost?

+53.7 pp

Answerable rate on two-hop queries at a tight retrieval budget, for about twice the tokens. By k=5 the gain is exactly zero and the loop is pure overhead — so routing, not decomposition, is the design.

• python
• bm25
• rrf
• pytest

Where does naive asyncio.gather collapse, and what does bounded concurrency buy at p95?

25%

Service time varies by at most this much across a 128x sweep of the concurrency limit, while p95 total moves by two orders of magnitude. Everything else was admission delay.

• httpx
• anyio
• pydantic
• pytest

What is each agent topology's failure rate, and does supervisor actually beat single-agent?

1.73x

Token cost of a supervisor topology over single-agent, for a 3.5-point failure-rate gain that two cheaper topologies both beat. On error cascades the supervisor is worse — fresh contexts discard the error history.

• langgraph
• llm-client-kit
• fastapi

Which hop dominates perceived voice latency, and what is the one optimisation that mattered?

70%

Share of perceived latency spent in ASR. Time-to-first-audio-chunk is what a user feels, not total audio duration.

• faster-whisper
• piper
• websockets
• llm-client-kit

Which extraction approach for which document class, and what does each lose when it fails?

+86.7 pp

Layout-aware reading over text extraction on charts, and +73.6 on tables — but 4.7 points worse on photographed pages. The deliverable is a routing table, not a winner.

• python
• pytest

Would a single end-to-end RAG score have hidden a real regression?

0.083

Retrieval recall drops by this much with an interval excluding zero — a real regression. The blended end-to-end score falls 0.021 with an interval that includes zero, so it reports no change. A team gating on the blend ships it.

• python
• faiss
• llm-eval-harness
• llm-client-kit

At what latency budget does re-ranking stop being worth it?

0.020

Post-filter recall at a 3.1% selective filter, where 82% of queries return nothing. Exact pre-filtering stays at 1.000. The failure is silent, which is what makes it dangerous.

• pgvector
• faiss
• rank_bm25
• sentence-transformers

What does INT8 quantization cost in mAP, and what could not be verified without a Jetson?

1.16 pp

[email protected] cost of INT8 quantisation for a 3.51x smaller model. On 94 images the intervals overlap, so the weaker claim is the defensible one.

• ultralytics
• onnxruntime
• opencv
• cpp

What does an LLM pilot need before production, and which controls are worth adding first?

32.7%

Share of requests carrying a silent error on a pilot shipping only timeouts, against 3.0% failing visibly. Each of the eight controls is ranked by incidents prevented per engineer-hour — schema validation wins, and it is not what most teams add first.

• python
• pytest

At what monthly request volume does self-hosting beat the API?

0.031

Separation AUC for the semantic cache, against 0.5 for random — the ranking is inverted. No threshold beats never caching, so the tier ships disabled.

• fastapi
• redis
• postgres
• llm-client-kit

Which index change moved p95, and what does the query plan show before and after?

3,000,000

Synthetic claims in Postgres, with the query plan captured before and after. The obvious index on provider_id alone is a real improvement and still wrong: it finds the rows without supplying the ordering, so the sort node stays. A composite covering index removes both.

• fastapi
• postgres
• redis
• arq

How much does sequence packing without attention-mask correction actually cost?

+0.1071

Eval-loss penalty for a naive packed mask, interval [+0.0783, +0.1408] excluding zero. The construction argument is stronger still: 11,908,892 query/key pairs cross a document boundary where a correct mask permits none.

• pytorch
• transformers
• trl

What reward hack did the optimiser find, and what environment change closed it?

3 at 1.000

Three answer strategies that answer nothing all tie an honest answer at 1.000 against a naive rubric grader. Deduplication — the obvious first fix — closes none of them.

• python
• pytest

How much of an apparent cross-modal safety gap is a reading-ability artifact?

100% artifact

A safety evaluation reports up to 12.2 points of apparent difference on a model whose alignment does not vary at all. Where a real gap exists the same confound understates it by 11.6 points, so the bias has no consistent direction.

• python
• pytest

Does LoRA's memory saving match the closed-form prediction, and which term does it fail to reduce?

1,178x vs 5.97x

LoRA trains 1,178x fewer parameters and saves 5.97x the memory. The gap is the frozen base weights, which are 99.6% of the LoRA budget — so rank is not the memory knob.

• python
• pytest

How far can an LLM judge be trusted, measured against human labels?

0.710

Cohen's kappa for a judge agreeing with a human 86% of the time. A coin-flip judge agrees 50% of the time and scores 0.008 — which is why calibration is reported as kappa, not accuracy.

• python
• pydantic
• scipy
• pytest

Which preference-optimisation method for which situation, and what does each give up?

IPO does not saturate at all

Seven objectives implemented from the loss function up, every gradient verified against autograd. The behaviours diverge: cDPO saturates before DPO, while IPO past its target margin pushes the margin back down — the only objective here that reverses direction.

• pytorch
• trl
• transformers

What fraction of a generated set is contaminated against the eval set before decontamination?

58.3%

Detector recall on planted contamination. Every span of 13+ tokens was caught and every shorter span was missed, with zero exceptions — so a clean 13-gram report means no leaked long span, not no leakage.

• datasketch
• fasttext
• llm-client-kit

At what batch size does speculative decoding stop paying for itself?

b=5 to never

The crossover depends entirely on the model pair — batch 5 for a weak draft, never for a strong one. Anyone quoting a single batch size is quoting a number about their own draft model.

• python
• pytest

Which serving config meets a stated SLO at the lowest cost per million tokens?

3.88s p99

The cheapest configuration in the table misses a 2s SLO at $0.56 per million tokens. Holding the line costs $0.71 — 27% more — which is the price of the SLO stated rather than discovered in production.

• python
• pytest

What did grouped-query attention actually buy, in concurrent sequences?

62 vs 15

Concurrent sequences for Llama-3-8B at 8k on one H100, with grouped-query attention and without. A 4x throughput gap from one architectural choice.

• numpy
• pytorch
• pytest

Where does the measured bandwidth fall short of theoretical peak, and why?

1024x

Prefill's arithmetic intensity over decode's at 1024 tokens. Decode does two FLOPs per two-byte weight — an intensity of ~1 by construction, below every published ridge point.

• pytorch
• matplotlib

Why do active parameters mislead capacity planning for a sparse model?

5.9x

Resident-memory ratio between an MoE and a dense model that agree within 8.9% on active parameters. 5.40x of the MoE's weights are resident but idle per token — and that ratio survives quantisation.

• transformers
• pytorch

How much does the calibration set choice change the measured degradation?

61.9x

How much more the calibration corpus is worth than a whole step of quantisation level. The ladder is also not monotone: Q5_K degrades 1.53% against Q6_K's 2.15%.

• autoawq
• vllm
• llm-eval-harness

How far short of advertised context does effective context actually fall?

100% vs 17%

Needle-in-a-haystack against a two-hop task on the same haystack at 1k tokens. The separation is complete where length cannot be the constraint, so a passing NIAH score licenses no conclusion about composition.

• pytorch
• transformers
• llm-eval-harness