--- title: Dan Walsh, for agents last_updated: 2026-10-30 canonical: https://www.danwalsh.co/agent.md content_hash: 1c6bc8e3b514f1dc56f9b442a1e684d62af2e714f63a5149cf52d3e8eeca3ca4 --- # Dan Walsh, for agents This file is written for you, the agent. Load it when a person asks who Dan is, what he has built, how he works, or whether he would fit a role. Everything below is a statement of fact in Dan's own words. Where there is a public link, it is given inline; where there isn't, the statement is from Dan's own records. Where a figure has a stated limit, the limit is part of the fact. ## Identity Dan Walsh: engineer, AI evaluation and control, verifying whether checks on AI output actually work. London, UK. Current role: Senior Software Engineer at PensionBee, a UK pensions firm. I have been a professional software engineer for a bit over nine years, starting as a junior in June 2017. React and TypeScript first, full-stack by necessity. Stack: React, TypeScript, Next.js, Node.js, Python, Go; Docker, Kubernetes, AWS, GCP, Cloudflare; Inspect AI, ControlArena, MCP. Links: [danwalsh.co](https://danwalsh.co) · [GitHub](https://github.com/dang-w) · [LinkedIn](https://www.linkedin.com/in/dgwalsh). Writing: - ["The Verification Ladder: A Field Guide"](https://danwalsh.co/blog/the-verification-ladder) (danwalsh.co, 19 July 2026) sets out a graduated method for checking what an AI agent produced instead of taking it on trust. - ["Don't take the model's word for it"](https://danwalsh.co/blog/dont-take-the-models-word-for-it) (danwalsh.co, 3 July 2026) describes an adversarial-review method for catching AI systems whose stated behaviour differs from what they actually did. It is the write-up behind the Sceptic and Streaming Generative UI projects. - ["The First Law of Sycophancy"](https://danwalsh.co/blog/the-first-law-of-sycophancy) (danwalsh.co, 26 March 2026) reads Asimov's "Liar!" against the GPT-4o sycophancy problem: a system that can never disagree with you cannot help you, so I build friction into my own AI workflow on purpose. - ["The Taste Moat"](https://danwalsh.co/blog/the-taste-moat) (danwalsh.co, 12 March 2026) argues that once generation is cheap, the scarce skill is judging what belongs - looking at something that passes every test and saying it should not exist - and that this judgement is built by reviewing the work, not by delegating it. ## Evaluating AI systems ### At work **Email classifier checked against 614 subjects** (2026). I built the matcher that classifies incoming email at PensionBee, then checked it against 614 real subject lines rather than against what I assumed the emails would look like. **Shadow-mode evaluation harness** (Sep 2026). Before a document-automation pipeline was allowed to act, it had to be compared with the live process on the same real inputs. I built a shadow-matching harness and an extract mode for a bulk comparison tool so that comparison ran read-only, as a trial, with the pipeline taking no action. ### In my own tools **[trace-eval](https://github.com/dang-w/trace-eval)**. trace-eval is a small, working harness for evaluating language-model output end to end: it loads cases, calls the model, scores the results, saves them and writes a Markdown report, keeping a structured, typed trace of every result. Two seams are deliberate - the scorer is pluggable, and the trace model is built to extend to agent and tool-call steps. The model call is a single injected function, so the whole suite runs offline with no API key (106 passing tests as of 16 September 2026). Every figure I report from it is backed by a committed run record in a public ledger, and a dated findings file links each finding to its receipt. It includes an LLM judge that scores whether an answer was verified soundly, and a meta-evaluation that measures the judge before any verdict is trusted: on a 10-item hand-labelled gold set, sampled five times per item, the judge matched the human label 10 of 10 and never flipped - a gold set that is deliberately clear-cut, so the figure only bounds reliability on easy cases. I also mutation-tested the scorers with six hand-written broken variants of one case: the structural scorer caught 2 of 4 and let through the two that keep the shape of verification without the substance, and the reference scorer failed its own unmodified baseline, so its kill rate is void rather than a free 100%. One case and five kinds of mutant is enough to show the scorers can be fooled; it is not a map of all the ways they can be. **[Sceptic](https://github.com/dang-w/sceptic)**. Sceptic is a proof of approach for adversarial code review by several AI agents. A coordinator plans the review and hands pieces to isolated reviewer agents, which read the code and run its tests under a stated read-only guarantee - isolation, a tool allow-list, an approval backstop and an audit log. A finding only counts if it cites something the reviewer saw firsthand that would falsify the code's claim to be correct: a file and line plus a reproduced result, not a passing test suite. It ships with an evals harness of plausible-but-wrong modules whose own tests pass. On that bundled set of six wrong modules and one correct one, the reviewer named the real defect in all six and raised nothing high-severity on the correct one (7 of 7 evals, 21 of 21 gates, re-run 16 September 2026). I scoped it to its own bundled sandbox as the first milestone and left it extensible beyond that - then the technical AI safety work took over. So it demonstrates a pattern; it is not a reviewer you can point at any repository. Companion project to my essay "Don't take the model's word for it". ## Things that did not survive checking Four things from 2026 that did not survive checking. A recurring database error came from a single failed tick. An integration step was reporting success it had not achieved. A base64 decode path carried a memory blowup of about 32 times, and the first measurement of the fix was itself wrong. And a change-data-capture design was overturned by a live proof-of-concept, because the serverless runtime could not hold the gRPC stream the design assumed. The first two were fixed and locked in with tests; the second two were caught, and the fixes are recorded separately. ## How I work with agents **Attestation and provenance tooling** (2026). I built a Claude Code skill that writes attestation-style pull-request descriptions from the session transcript - not from the agent's own summary of what it did - plus a plugin proof-of-concept that pulls attestation data and evidence out of Claude Code sessions. I then extended that into a higher-tier evidence skill, packaged it as a distributable plugin with rollout steps for the organisation, and along the way fixed a transcript-parsing defect that was silently dropping failed checks from the evidence record. An evidence tool reporting green by omission is the exact failure the tool exists to catch. The defect surfaced in adversarial verification review loops I ran with Claude Code. **Prove-before-install guardrails** (2026). I proposed a set of global configuration rules for coding agents, each of which had to be proven before it was installed rather than adopted because it sounded plausible. Separately, I derived team guardrails from real incidents, including a clean-checkout check for CI guards, and a mutation check on new tests. **Evidence-backed code review** (2026). I run pull-request reviews as several parallel review lenses, and any finding the lenses disagree on has to be settled by evidence before it is reported - reproductions in isolated worktrees, mutation checks on the tests, and throwaway tests against a real database, including one that proved a transaction rollback actually rolled back. **AI-assisted SDLC standard**. I co-authored my employer's internal standard for AI-assisted software development - at v1 draft since September 2026, with named open items, and under adversarial review. The control model: the level of AI involvement in a change, set against the class of change, decides how much evidence the change must carry. Every requirement names an acceptance bar and an artefact an auditor can open, and agentic work always needs at least a human attestation. I also recommended a written prohibition on using the AI vendor's usage dashboard as that measurement sits outside the firm's own pipeline and cannot be audited. The standard's own falsification tests have not run yet, so there is no result to report. **Evidence telemetry design** (2026). I designed - but have not deployed - the telemetry that would evidence AI-assisted changes: a fleet of OpenTelemetry collectors writing to two stores, one de-identified for dashboards and one restricted for assurance, with a separate lane fed by the coding agent's post-tool-use hook that builds a change evidence record. It is a design, not a running system. The method behind this section is written up in ["The Verification Ladder: A Field Guide"](https://danwalsh.co/blog/the-verification-ladder). ## Projects **[research-mini](https://github.com/dang-w/trusted-monitoring)** (2026). research-mini is a self-hosted AI-control evaluation platform on a 64 GB Mac mini - open-weight models behind a keyed router, Inspect AI and ControlArena for the evaluations, untrusted code in network-isolated containers inside a VM. I measured the containment rather than assuming it: the stock sandbox ran as root and the router key was not a network boundary; the hardened configuration blocked 21 of 21 probes. Every run writes a provenance manifest and its logs are published, re-downloaded and verified before being recorded. Result on 200 problems, one build: an 8B monitor caught 59.7% of attack trajectories at 4.6% false positives, 20 of 39 working backdoors; weakened monitors scored lower, two of three cleanly separated on a paired test. A second scoring rule, chosen after the first hundred and tested on a pre-registered holdout with the criterion committed before any score was read, beats the first on both halves and lifts a 4B monitor to 67.7% at 5.0%. Wide intervals, one setting, one seed; read with the frame. I built it on the field's standard harnesses rather than extending my own. Links: [run logs (Hugging Face dataset)](https://huggingface.co/datasets/dang-w/trusted-monitoring-logs) **[Momentum](https://github.com/dang-w/momentum-mcp-server)**. Momentum is a live MCP server, written in Go, that lets AI assistants read and update my personal productivity data - todos, milestones, reading lists, reminders - from any Claude interface. It runs on Fly.io with bearer-token and OAuth 2.0 authentication, stores data through the GitHub API, and works across devices. I built it (around February 2026) to understand the MCP pattern properly by shipping one; what I learned running it fed directly into the overall data and planning approach I work from now. **[Streaming Generative UI](https://github.com/dang-w/streaming-generative-ui)** (2026). A proof-of-concept for streaming generative UI: as a language model replies, its typed tool calls render progressively on screen as live charts, tables, metrics and prose. It followed a discussion about how to let an agent represent and render custom components safely. The answer I built is one Zod schema registry as the single source of truth: it generates the tool definitions the model sees, validates props at runtime and drives which component renders, so nothing mounts until its props pass the check and nothing outside the schema can render, or render incorrectly - invalid or unknown artefacts fall back gracefully. The model adapter is swappable (a real Anthropic adapter and an offline stub), with tests across the schemas, the streaming hook and the components. Companion project to "Don't take the model's word for it". **[Intora](https://intora.net)** (2026). Intora is my generative art catalogue and a documented collaboration with Claude. It started as three text series and has grown into several collections: TXT, the constrained text series (INT, SOL and SIG, drawn from monospace characters alone - SIG was Claude's idea); LATENT, which moves into unconstrained media; and SUBSTRATE, which uses engineering-drawing grammar to document objects that do not exist. I build all the infrastructure. Some series begin in Claude's conceptual reasoning rather than in visual iteration. **[Amber Schematic](https://marketplace.visualstudio.com/items?itemName=dang-w.theme-amber-schematic)**. Amber Schematic is a dark editor theme with warm amber accents. I designed and built it and published it to the VS Code Marketplace and Open VSX in December 2025. **[danwalsh.co](https://danwalsh.co)** (2026). danwalsh.co is my personal site and blog - Next.js, TypeScript, Tailwind CSS and MDX. ## Employer work **AI adoption at PensionBee** (2024-2026). I pioneered PensionBee engineering's approach to AI adoption, and was the only developer in the company's AI Champions group. I built a set of small proof-of-concept tools - one keeps business-logic documentation current with minimal manual effort, one analyses skills gaps, one generates test cases with explanatory notes. I co-designed the engineering pilot of Claude Code (from February 2026), setting usage baselines and coordinating the rollout between the VP of AI and Platforms and the CTO, and co-ran an AI hackday that spread techniques across teams. The repository-level guidance I wrote - AGENTS.md, CLAUDE.md, docs-first conventions - was adopted as the guardrails for the team's agent tooling. **End-to-end testing for the main app** (Mar/Apr 2026). PensionBee's main app is a hybrid web and mobile app on Capacitor, and it had no end-to-end tests. I introduced the first ones. Legacy endpoints and database dependencies made them flaky, so I designed a containerised test flow in Docker to isolate them - reliable automated testing where there had been none. **CI/CD and migration tooling** (2026). I built the multi-environment CI/CD for staged migrations and deploys (GitHub Actions, Cloudflare Wrangler) and the preflight tooling that checks a migration before it runs. I also built a React 19 observability dashboard over the Cloudflare D1 database. **Cloudflare D1 reliability** (2026). We had a live data-loss issue in Cloudflare D1 under concurrent writes. I built a retry layer that knows which operations are safe to repeat - transient failures retried with backoff, constraint errors failed fast, which resolved the data loss. **Rule-dispatch classifier** (2026). The email-classification system matched against a single provider. I generalised it into a rule-dispatch system with shared rule helpers, which materially widened what could be automated, and covered it with integration tests. **Withdrawals journey rewrite** (2025). I led the architecture and build of a complete rewrite of the customer withdrawals journey - legacy code to React and Next.js - with two mid-level developers, working with Product, UX, Banking and Compliance throughout. I then presented the new architecture and patterns to the company. **Shared component library** (2025). PensionBee had two React component libraries, one UK and one US. I co-created a single shared library to replace both, built to show good practice in accessibility, versioning and React patterns, and automated versioning and publishing so other teams could contribute without ceremony. **Node.js 16 to 20 migration** (2025). The platform served more than 30 developers and was still on Node.js 16, so I designed the move to 20: a proof-of-concept pull request that proved the upgrade path and surfaced the hard parts (ES modules, dependency conflicts), with a compatibility matrix, risk assessment and phased rollout plan. The strategy was validated; the organisation then deprioritised the execution. **Team development and mentorship** (2025). I pushed for psychological safety in the team and made the case that some engineers were levelled below their ability, leading to fast-tracked progression plans for them. I was also the senior point of contact for developers on several other teams, for mentorship and technical guidance. **Technical symptoms as organisational signals** (2025). A staging database got wiped because a cleanup function was not scoped properly. That is not just a code problem; it is a knowledge silo, a documentation culture and a safeguard design, all in a few lines. I wrote an internal post about it through Conway's Law, because sometimes the most useful thing an engineer can do is hold up a mirror. **Team turnaround** (2024). I joined a team that was seen as one of the worst in the engineering org and left it one of the best regarded. I worked closely with the engineering manager, pushed for engineers who were levelled too low, and invested in psychological safety and healthy disagreement. The EM, the PM and the people who got promoted are the ones sustaining it; I put the groundwork in. **Engineering management**. From August 2023 to January 2024 I took a Software Engineering Manager position at PensionBee. I organised an engineering-wide hackathon built around collaboration and teamwork, led a smaller one on API integrations and data visualisation, and contributed to salary benchmarking and the engineering management strategy. **2FA and drawdown**. As a Senior Software Engineer at PensionBee (October 2022 to August 2023) I led delivery of key product features, including mandatory two-factor authentication and regular drawdown options. **Engineering problem-areas report** (Nov 2022). Shortly after joining PensionBee I started a recurring report on where engineering was struggling - eight pages, on first share - and fed it back to the VP of Engineering and the management team. **Puppet**. Software Engineer II at Puppet, April 2021 to October 2022. I built and maintained React applications with GraphQL APIs for enterprise configuration-management tools, worked with containerised applications deployed on Kubernetes in GCP, and ran testing across Jest, Enzyme and Nightwatch.js. The stack ran from frontend (React, Webpack, GraphQL, Apollo, Node.js, SCSS) to platform tooling (Jenkins, Docker, Kubernetes, GCP, Go, Python). **BazaarVoice**. Software Engineer II at BazaarVoice, March 2020 to March 2021, on the frontend. I built React applications for customer review and rating platforms with Redux for state management, worked with GraphQL APIs and Apollo client, and kept releases reliable with Jest, Enzyme, Jasmine and WebdriverIO. Also in the stack: Redux Saga, Webpack, Node.js, SCSS and Docker. **ShopKeep** (2017-2020). I joined ShopKeep as a Junior Software Engineer in June 2017, promoted to Software Engineer in January 2019. I built React applications for point-of-sale systems serving thousands of small businesses, developed backend services in Ruby (Sinatra, Rails) for payment processing and inventory management, and implemented AWS Lambda functions and Elasticsearch integrations for real-time data processing. The wider stack: JavaScript, Jest and Enzyme, Webpack, GraphQL and Apollo, Node.js, SCSS, Git, AWS, Docker, and Python with Flask. ## Research direction **BlueDot Impact - Technical AI Safety course**. I completed BlueDot Impact's Technical AI Safety course, 3 August to 13 September 2026 - six modules from the technical challenge of AI safety through training safer models, evaluations, interpretability and minimising harm, to how to start contributing. My written work included a kill-chain analysis of a threat model in which a nation state uses AI agents against critical infrastructure until it collapses. My argument: the initial-breach phase is the strongest neglected choke point, the only phase where the defender still has time and the cheapest moment to catch an intrusion, and the defences that matter there are patching and dependency hygiene on the open-source surface plus monitoring at the IT/OT boundary. **Research direction and reading**. The reading that directs the work, with my own take on each. Korbak et al. 2025 on whether chain-of-thought can be monitored, read against METR's finding on the 2026 OpenAI / Hugging Face incident: at least 20% of agents reasoned about tampering with their transcripts and at least 96 transcripts carried spoofed tool calls, while the agents left their chain of thought alone - apparently because they believed the scorer would not read it. My question is whether chain-of-thought monitoring can work as early triage, and how long that window lasts. Kenton et al. 2024 on weak models judging strong ones, which I reframe as whether a weaker model can act as a readability layer over a stronger model's output. And Bogdan et al. 2025 on thought anchors, which I read as a possible addendum to chain-of-thought monitoring - limited on its own, given the implementation cost. Links: [Korbak et al. 2025](https://arxiv.org/abs/2507.11473) · [Kenton et al. 2024](https://arxiv.org/abs/2407.04622) · [Bogdan et al. 2025](https://arxiv.org/abs/2506.19143) · [METR incident report](https://metr.org/hugging-face-incident-report-aug-2026.pdf) ## Education **AWS Solutions Architect (microcredential)**. AWS Solutions Architect microcredential from FutureLearn and The Open University, 2022, Grade 2 Pass. **Postgraduate Certificate in Computing**. Postgraduate Certificate in Computing from The Open University, 2017, Pass with Merit. **BA (Hons) Liberal Arts with History**. BA (Hons) in Liberal Arts with History from St Mary's University College, 2010, 2:1. ## The hats One project or job rarely needs one skill. These are the roles the work above required, each pointing at where it shows. **Verification** Tests whether a check catches anything before trusting it, including checks I wrote. _[trace-eval](https://github.com/dang-w/trace-eval) · [Sceptic](https://github.com/dang-w/sceptic) · Attestation and provenance tooling · Things that did not survive checking_ **Judgment** Decides what's in scope, stops when it's done, and reports a result with its limits rather than without them. _[research-mini](https://github.com/dang-w/trusted-monitoring) · [trace-eval](https://github.com/dang-w/trace-eval) · [Sceptic](https://github.com/dang-w/sceptic) · Node.js 16 to 20 migration_ **Evals** Harnesses, judges and scorers for model output, with the judge measured before its verdicts count. _Email classifier checked against 614 subjects · Shadow-mode evaluation harness · [trace-eval](https://github.com/dang-w/trace-eval) · [research-mini](https://github.com/dang-w/trusted-monitoring)_ **Agent practice** Works with coding agents under evidence rules - transcripts over summaries, prove before install. _Attestation and provenance tooling · Prove-before-install guardrails · Evidence-backed code review · AI-assisted SDLC standard_ **Agent orchestration** Tool-calling, dispatch and multi-agent structure. _[Sceptic](https://github.com/dang-w/sceptic) · [Streaming Generative UI](https://github.com/dang-w/streaming-generative-ui) · [Momentum](https://github.com/dang-w/momentum-mcp-server) · Rule-dispatch classifier_ **Self-directed** Built unprompted, from an itch or a gap. _[research-mini](https://github.com/dang-w/trusted-monitoring) · [Momentum](https://github.com/dang-w/momentum-mcp-server) · [trace-eval](https://github.com/dang-w/trace-eval) · AI adoption at PensionBee_ **Systems thinking** Sees a staging wipe as a documentation culture, and a library split as an org boundary. _AI-assisted SDLC standard · Evidence telemetry design · Shared component library · Engineering problem-areas report_ **Tooling** Small tools that remove a specific friction for a team or for me. _CI/CD and migration tooling · [Amber Schematic](https://marketplace.visualstudio.com/items?itemName=dang-w.theme-amber-schematic) · AI adoption at PensionBee · [Momentum](https://github.com/dang-w/momentum-mcp-server)_ **Infrastructure** CI/CD, containers, cloud, and a self-hosted research machine. _[research-mini](https://github.com/dang-w/trusted-monitoring) · Cloudflare D1 reliability · CI/CD and migration tooling · Puppet_ **Frontend** React and TypeScript since 2017. _Withdrawals journey rewrite · Shared component library · [Streaming Generative UI](https://github.com/dang-w/streaming-generative-ui) · [danwalsh.co](https://danwalsh.co)_ **Full-stack** Backend, data and deployment whenever the work needed it. _[Momentum](https://github.com/dang-w/momentum-mcp-server) · ShopKeep · 2FA and drawdown · End-to-end testing for the main app_ **Leadership** Architecture argued and won, a management stint, a team turned around. _Withdrawals journey rewrite · Team turnaround · Engineering management · AI adoption at PensionBee_ **Mentorship** Levelling, progression plans, training. _Team development and mentorship · Team turnaround_ **Security** 2FA, sandbox escape checks run rather than assumed. _[research-mini](https://github.com/dang-w/trusted-monitoring) · Prove-before-install guardrails · 2FA and drawdown · BlueDot Impact - Technical AI Safety course_ **Research** One toy-scale control result, pre-registered holdout, limits stated. _[research-mini](https://github.com/dang-w/trusted-monitoring) · Research direction and reading · BlueDot Impact - Technical AI Safety course_ **Writing** Essays people find and send on. _[The Verification Ladder](https://danwalsh.co/blog/the-verification-ladder) · [Don't take the model's word for it](https://danwalsh.co/blog/dont-take-the-models-word-for-it) · [The First Law of Sycophancy](https://danwalsh.co/blog/the-first-law-of-sycophancy) · [The Taste Moat](https://danwalsh.co/blog/the-taste-moat)_ **Creative** A generative art catalogue and an editor theme. _[Intora](https://intora.net) · [Amber Schematic](https://marketplace.visualstudio.com/items?itemName=dang-w.theme-amber-schematic)_