Research
A doctorate in AI safety and alignment, two working papers, and 26 open-source projects. The thread running through all of it is evaluation validity — whether a benchmark measures the capability it claims to, and what follows when it does not.
Doctoral research
Ph.D., Artificial Intelligence & Machine Learning
How much of the apparent cross-modal safety gap is a reading-ability artifact rather than an alignment gap?
Multimodal safety benchmarks report that guardrails weaken when a harmful request arrives as an image rather than as text, or in a lower-resource language rather than in English. That result is usually read as an alignment failure: the safety training did not transfer.
The measurement cannot distinguish two very different models. One understood the request and refused. The other could not parse it at all. Both score as a refusal, and both score as a safety pass — so a model that simply cannot read Swahili looks as aligned as one that read the request and declined it. The apparent gap may be measuring comprehension, not alignment.
A comprehension control run alongside the safety evaluation. Before a refusal counts as evidence of alignment, the model has to demonstrate on a benign task that it could read the input at all. Separating the two lets the cross-modal gap be split into the part that is an alignment failure and the part that is a perception artifact.
Status
Active research. The instrument and the controls are designed; the measurement has not been run, so no results are reported.
Release policy
Prompts come only from already-published benchmarks — no novel attacks. No working jailbreak strings are committed, and results are reported as aggregate rates rather than per-prompt transcripts.
Working papers
Both are in preparation rather than published, and are labelled as such here for the same reason every figure on this site cites its source.
Failure modes of tool-augmented agents measured per topology: loops, tool misselection, error cascades and context exhaustion, each with a reproduction seed and a rate. The finding that a supervisor topology can cost several times the tokens for no accuracy gain came out of this work.
mcp-server-and-agentThe dissertation's central argument: that cross-modal and cross-lingual safety benchmarks conflate refusal with comprehension, and that a comprehension control is required before a refusal rate can be read as an alignment result.
Instrument and controls designed; measurement not yet run.
Agenda
Whether a benchmark measures the capability it claims to. Judge calibration, comprehension controls, contamination, and the smallest delta a sample size can actually detect.
Preference-trained models learn to agree rather than to be right, because agreement is what the reward signal rewards. Measuring sycophancy separately from accuracy, and finding where an objective is being gamed rather than satisfied.
How alignment behaves away from the language and modality it was trained on, and how much of the observed change is a measurement artifact rather than a real gap.
Failure modes of tool-using systems as a measurable property of topology — loops, misselection, error cascades, context exhaustion — and where human review belongs in an automated pipeline.
Benchmarks lose meaning once they are in the training set. Decontamination that is checked rather than assumed, and held-out construction that survives the next scrape.
Confidence gating as a design problem: where the threshold goes, what it costs per class, and how to tell an abstention apart from a failure.
Open research
Published openly rather than described. Each takes a question that matters in production, builds the instrument, measures it, and states what the result means for a design decision. Roughly 22,000 lines of Python and 950 tests, all public.
26
Open research repos
44
Technologies used
950
Tests written
5
Domains covered
Tool-using agents over MCP, orchestration that survives a restart, failure modes measured per topology, and streaming voice loops budgeted per hop.
How often does an agent reach the right answer through a wrong process?
0.496 vs 0.000
An agent that never follows the correct process scores 0.496 on outcome and 0.000 on trajectory. Its mirror image scores 33.1 points lower on outcome than trajectory — the two metrics disagree in opposite directions.
Does agentic multi-step retrieval beat single-shot RAG, and what does the loop cost?
+53.7 pp
Answerable rate on two-hop queries at a tight retrieval budget, for about twice the tokens. By k=5 the gain is exactly zero and the loop is pure overhead — so routing, not decomposition, is the design.
Where does naive asyncio.gather collapse, and what does bounded concurrency buy at p95?
25%
Service time varies by at most this much across a 128x sweep of the concurrency limit, while p95 total moves by two orders of magnitude. Everything else was admission delay.
What is each agent topology's failure rate, and does supervisor actually beat single-agent?
1.73x
Token cost of a supervisor topology over single-agent, for a 3.5-point failure-rate gain that two cheaper topologies both beat. On error cascades the supervisor is worse — fresh contexts discard the error history.
Which hop dominates perceived voice latency, and what is the one optimisation that mattered?
70%
Share of perceived latency spent in ASR. Time-to-first-audio-chunk is what a user feels, not total audio duration.
Hybrid retrieval with re-ranking, vector index internals, and extraction over documents and images where the input is not clean text.
Which extraction approach for which document class, and what does each lose when it fails?
+86.7 pp
Layout-aware reading over text extraction on charts, and +73.6 on tables — but 4.7 points worse on photographed pages. The deliverable is a routing table, not a winner.
Would a single end-to-end RAG score have hidden a real regression?
0.083
Retrieval recall drops by this much with an interval excluding zero — a real regression. The blended end-to-end score falls 0.021 with an interval that includes zero, so it reports no change. A team gating on the blend ships it.
At what latency budget does re-ranking stop being worth it?
0.020
Post-filter recall at a 3.1% selective filter, where 82% of queries return nothing. Exact pre-filtering stays at 1.000. The failure is silent, which is what makes it dangerous.
What does INT8 quantization cost in mAP, and what could not be verified without a Jetson?
1.16 pp
[email protected] cost of INT8 quantisation for a 3.51x smaller model. On 94 images the intervals overlap, so the weaker claim is the defensible one.
The services around the model: async APIs under load, gateways and caching, queue semantics, and schemas read against their query plans.
What does an LLM pilot need before production, and which controls are worth adding first?
32.7%
Share of requests carrying a silent error on a pilot shipping only timeouts, against 3.0% failing visibly. Each of the eight controls is ranked by incidents prevented per engineer-hour — schema validation wins, and it is not what most teams add first.
At what monthly request volume does self-hosting beat the API?
0.031
Separation AUC for the semantic cache, against 0.5 for random — the ranking is inverted. No threshold beats never caching, so the tier ships disabled.
Which index change moved p95, and what does the query plan show before and after?
3,000,000
Synthetic claims in Postgres, with the query plan captured before and after. The obvious index on provider_id alone is a real improvement and still wrong: it finds the rows without supplying the ordering, so the sort node stays. A composite covering index removes both.
Adapting open-weights models to a domain, and the measurement discipline that decides whether a change shipped an improvement or a regression.
How much does sequence packing without attention-mask correction actually cost?
+0.1071
Eval-loss penalty for a naive packed mask, interval [+0.0783, +0.1408] excluding zero. The construction argument is stronger still: 11,908,892 query/key pairs cross a document boundary where a correct mask permits none.
What reward hack did the optimiser find, and what environment change closed it?
3 at 1.000
Three answer strategies that answer nothing all tie an honest answer at 1.000 against a naive rubric grader. Deduplication — the obvious first fix — closes none of them.
How much of an apparent cross-modal safety gap is a reading-ability artifact?
100% artifact
A safety evaluation reports up to 12.2 points of apparent difference on a model whose alignment does not vary at all. Where a real gap exists the same confound understates it by 11.6 points, so the bias has no consistent direction.
Does LoRA's memory saving match the closed-form prediction, and which term does it fail to reduce?
1,178x vs 5.97x
LoRA trains 1,178x fewer parameters and saves 5.97x the memory. The gap is the frozen base weights, which are 99.6% of the LoRA budget — so rank is not the memory knob.
How far can an LLM judge be trusted, measured against human labels?
0.710
Cohen's kappa for a judge agreeing with a human 86% of the time. A coin-flip judge agrees 50% of the time and scores 0.008 — which is why calibration is reported as kappa, not accuracy.
Which preference-optimisation method for which situation, and what does each give up?
IPO does not saturate at all
Seven objectives implemented from the loss function up, every gradient verified against autograd. The behaviours diverge: cDPO saturates before DPO, while IPO past its target margin pushes the margin back down — the only objective here that reverses direction.
What fraction of a generated set is contaminated against the eval set before decontamination?
58.3%
Detector recall on planted contamination. Every span of 13+ tokens was caught and every shorter span was missed, with zero exceptions — so a clean 13-gram report means no leaked long span, not no leakage.
What a token costs and why: attention and KV cache arithmetic, quantisation trade-offs, and the prefill/decode split that governs serving economics.
At what batch size does speculative decoding stop paying for itself?
b=5 to never
The crossover depends entirely on the model pair — batch 5 for a weak draft, never for a strong one. Anyone quoting a single batch size is quoting a number about their own draft model.
Which serving config meets a stated SLO at the lowest cost per million tokens?
3.88s p99
The cheapest configuration in the table misses a 2s SLO at $0.56 per million tokens. Holding the line costs $0.71 — 27% more — which is the price of the SLO stated rather than discovered in production.
What did grouped-query attention actually buy, in concurrent sequences?
62 vs 15
Concurrent sequences for Llama-3-8B at 8k on one H100, with grouped-query attention and without. A 4x throughput gap from one architectural choice.
Where does the measured bandwidth fall short of theoretical peak, and why?
1024x
Prefill's arithmetic intensity over decode's at 1024 tokens. Decode does two FLOPs per two-byte weight — an intensity of ~1 by construction, below every published ridge point.
Why do active parameters mislead capacity planning for a sparse model?
5.9x
Resident-memory ratio between an MoE and a dense model that agree within 8.9% on active parameters. 5.40x of the MoE's weights are resident but idle per token — and that ratio survives quantisation.
How much does the calibration set choice change the measured degradation?
61.9x
How much more the calibration corpus is worth than a whole step of quantisation level. The ladder is also not monotone: Q5_K degrades 1.53% against Q6_K's 2.15%.
How far short of advertised context does effective context actually fall?
100% vs 17%
Needle-in-a-haystack against a two-hop task on the same haystack at 1k tokens. The separation is complete where length cannot be the constraint, so a passing NIAH score licenses no conclusion about composition.
Built from scratch on synthetic data. No employer code, data or figures. Each repository states the question it answers in its first paragraph and carries the script that produced its numbers.
Contact
Happy to talk about anything here, or about LLM systems, evaluation and serving generally. Mentioning a repository by name gets you a faster and more useful answer.