# Ishan Avasthi > AI Engineer — I build LLM-powered systems, agentic workflows, and developer tools. The complete portfolio of Ishan Avasthi in one document: bio, skills, every project in full, and every blog post in full. The short index is at https://ishanavasthi.in/llms.txt. ## About I'm a CS undergrad at BITS Pilani working on AI engineering. I build the layer between raw language models and working products: agent pipelines, RAG systems, and the observability and evals that make them reliable enough to ship. In practice that means pipelines that retrieve the right context, agents that take actions, and tooling that makes the whole thing reliable. I care about testing things properly, making systems easy to work with, and shipping stuff that holds up in the real world. - **Role:** AI Engineer - **Based in / studying at:** IND · BITS Pilani - **Status:** open to AI engineering roles - **Email:** hello@ishanavasthi.in - **Website:** https://ishanavasthi.in - **GitHub:** https://github.com/ishanavasthi - **LinkedIn:** https://www.linkedin.com/in/ishanavasthi ## Focus areas - **Agents & LLM Systems** (12 projects): Multi-agent pipelines, tool-calling agents, and the observability around them. - **RAG & Document AI** (3 projects): Retrieval pipelines over documents people actually need to read. - **RL Environments & Evals** (2 projects): OpenEnv-compliant environments for training and grading agents. - **Developer Tools** (3 projects): Small, sharp utilities built to remove a specific daily friction. - **Full-Stack & Web** (4 projects): Product-shaped web apps, end to end. - **Foundations** (9 projects): Earlier work — Spring Boot services, low-level design, Java, and data analysis. ## Skills - **Agents & LLM Systems** (the core focus): Multi-agent pipelines, MCP, Tool calling, Claude API, Gemini, Groq, LangGraph, LangChain, Voice agents · LiveKit, RAG · ChromaDB — 15 projects - **Evals & Reliability** (what makes agents shippable): Agent evals, OpenTelemetry tracing, LangSmith, RL environments · OpenEnv, CI regression gates, Deterministic guardrails, Prompt-injection defense, SigNoz / ClickHouse — 9 projects - **Languages** (daily drivers first): Python, TypeScript, JavaScript, Java, SQL, C# — 14 Python, 13 TS - **Backend & Infra**: FastAPI, Node / Express, Spring Boot, Postgres, Supabase, MongoDB, Docker, GitHub Actions, AWS, asyncio / TCP — 15 projects - **Frontend**: Next.js, React, Tailwind CSS, React Native, Chrome Extensions · MV3 — 9 projects _33 projects. Commit activity synced from GitHub on 2026-09-03._ ## Projects: Agents & LLM Systems Multi-agent pipelines, tool-calling agents, and the observability around them. ### Agent Store _Merchant-side agentic commerce: messy captions in, buyable MCP catalog out._ Turns long-tail sellers' Instagram and Hinglish captions into an AI-buyer-readable catalog over MCP plus a REST twin, via server-side extraction with a 0.90 confidence gate that never invents stock. Purchases run a signed intent → cart → payment mandate chain with per-agent caps and per-purchase budgets on real Razorpay test rails; every decision lands in a hash-chained, append-only audit log replayable in the viewer and independently re-checked. 29 products and 92 variants from 28 hand-labelled captions, with fail-closed decline and auto-refund oversell rehearsals. - **Year:** 2026 (featured) - **Tags:** TypeScript, MCP, Claude, Multi-Agent, Evals, Full-Stack - **Tech:** TypeScript, Express, MCP SDK, Drizzle, Neon Postgres, Razorpay, React, Vite, Vitest, Railway - **Source:** https://github.com/ishanavasthi/agent-store - **Live / demo:** https://link.ishanavasthi.in/agent-store - **Activity:** 151 commits, 2026-08-21 → 2026-09-03 ### Preflight _CI gate that catches AI agent regressions before they ship._ Built a CI tool that catches cost, latency, token, and success-rate regressions in AI agent systems before deployment, comparing agent behavior across commits and posting findings to pull requests with deep links to the offending trace spans. A diagnosis agent queries SigNoz over MCP to explain each failed gate in natural language, and its own investigation is traced and queryable. - **Year:** 2026 (featured) - **Tags:** Python, Claude, MCP, Docker, Multi-Agent, Evals, DevTools - **Tech:** Python, OpenTelemetry, SigNoz, ClickHouse, MCP, Claude API, Docker Compose, GitHub Actions, YAML - **Source:** https://github.com/ishanavasthi/preflight - **Live / demo:** https://link.ishanavasthi.in/preflight-video - **Activity:** 33 commits, 2026-07-26 → 2026-07-27 ### ClinicFlow _Real-time AI voice receptionist for clinics, with a live operator dashboard._ Built a real-time AI voice agent that answers patient calls, runs intake, books appointments, and routes callers, with a live operator dashboard. Combines a barge-in capable LiveKit pipeline with auditable LLM function tools and deterministic guardrails so the agent never invents facts or skips steps. - **Year:** 2026 (featured) - **Tags:** Python, TypeScript, LiveKit, FastAPI, Groq, Next.js, Voice AI, Full-Stack - **Tech:** Python, LiveKit, Deepgram STT, Groq, Numik TTS, FastAPI, SQLModel, SQLite, Next.js, TypeScript - **Source:** https://github.com/ishanavasthi/clinicflow - **Live / demo:** https://link.ishanavasthi.in/clinicflow-video - **Activity:** 45 commits, 2026-07-18 → 2026-08-06 ### AgentGrid _Nine Gemini agents take an issue from bug report to merged PR, unattended._ Built a coding pipeline of 9 Gemini agents that takes a software issue from report to tested, merged pull request with zero human intervention. Specialized agents (Planner, Coder, Reviewer, Integrator, Tester, Publisher) collaborate through context-preserving handoffs and conflict resolution, backed by a live dashboard. - **Year:** 2026 (featured) - **Tags:** Python, Gemini, Multi-Agent, DevTools - **Tech:** Python, Gemini API, google-genai, SSE, subprocess/git, threading, Playwright, HTML/CSS/JS - **Source:** https://github.com/ishanavasthi/agentgrid - **Live / demo:** https://link.ishanavasthi.in/agentgrid-demo-video - **Activity:** 12 commits, 2026-07-11 → 2026-07-16 ### AlphaDesk _Multi-tenant portfolio analyzer for Indian investors, narrated by agents that can't invent a number._ Sign in with Clerk, link an IND Money account over MCP, and get net worth, allocation, holdings, and a nightly net-worth trend, with an AI overview written by a fan-out of specialist agents. Every figure is computed in Python first, and any digit an agent writes freehand trips a scripted fallback, so the narrative can never fabricate one. Per-user broker tokens are Fernet-encrypted in Neon Postgres; a labelled paper-trading Lab runs the multi-agent research desk as a simulation, never mixed with real holdings. - **Year:** 2026 (featured) - **Tags:** Python, TypeScript, LangGraph, FastAPI, Next.js, MCP, Multi-Agent, Full-Stack - **Tech:** LangGraph, FastAPI, Next.js, TypeScript, Clerk, Neon Postgres, SQLModel + Alembic, IND Money MCP, OpenAI, Groq - **Source:** https://github.com/ishanavasthi/alphadesk - **Live / demo:** https://link.ishanavasthi.in/alphadesk - **Activity:** 240 commits, 2026-06-24 → 2026-08-22 ### Swiggy Claw _Order food, groceries, and tables in plain English over Swiggy's MCP platform._ Built a full-stack AI ordering agent for food, groceries, and restaurant bookings using plain English, powered by Swiggy's MCP platform, Groq tool calling, resilient retries, and a streaming Next.js chat UI. - **Year:** 2026 (featured) - **Tags:** TypeScript, Next.js, MCP, Groq, Multi-Agent, Full-Stack - **Tech:** Next.js, TypeScript, Tailwind CSS, MCP, Groq, OpenAI Tool Format - **Source:** https://github.com/ishanavasthi/swiggy-claw - **Live / demo:** https://link.ishanavasthi.in/swiggy-claw - **Activity:** 21 commits, 2026-05-31 → 2026-07-30 ### ReviewGrid _Agent-callable code review skill returning a deterministic verdict._ Built a backend code-review skill any agent can call: POST a raw diff or a PR URL and get back a deterministic pass / needs_changes / fail verdict with line-anchored issues. Runs as a Deno service on Base44 with static analysis behind the verdict, so the same diff always grades the same way. - **Year:** 2026 (featured) - **Tags:** TypeScript, Deno, Claude, DevTools, Evals - **Tech:** TypeScript, Deno, Base44, Static Analysis, Agent Skills - **Source:** https://github.com/ishanavasthi/reviewgrid - **Live / demo:** https://reviewgrid-915b1bbc.base44.app - **Activity:** 18 commits, 2026-07-28 → 2026-08-03 ### Panorama _Reviews a pull request using evidence from the other repos in the org._ Built a CLI and watcher that reviews a GitHub PR against its sibling repositories — a rename that breaks a client, a helper that already exists next door, an endpoint that ignores an org convention — citing repo/path:line and never copying source across a trust boundary. Four independent retrieval channels (lexical, dependency graph, symbol index, HTTP contract) are rank-fused to orient the model, then every citation it returns is re-checked against the real files, so an unsupported finding is discarded rather than downgraded. V2 turned quality into a number gated on every commit: an 18-case labelled corpus over six repos and three languages, recall@3 of 1.000, and zero false positives on negative controls across 54 live runs. - **Year:** 2026 - **Tags:** Python, Claude, DevTools, Evals - **Tech:** Python, Typer, Pydantic, uv, Claude Code CLI, GitHub CLI, git - **Source:** https://github.com/ishanavasthi/panorama/tree/v2 - **Live / demo:** https://link.ishanavasthi.in/panorama-video - **Activity:** 25 commits, 2026-08-06 → 2026-08-09 ### PR Review Bot _GitHub App that reviews every pull request with Gemini._ Built a GitHub App that autonomously reviews pull requests using LLMs and posts structured feedback directly on the PR. On open, sync, or reopen it fetches the diff, sends it to Gemini for analysis, and comments a structured review. - **Year:** 2026 - **Tags:** TypeScript, Gemini, Docker, DevTools - **Tech:** TypeScript, Node.js, Express.js, Octokit, Docker, GitHub Actions, Gemini API - **Source:** https://github.com/ishanavasthi/pr-review-bot - **Activity:** 1 commits, 2026-03-19 → 2026-03-19 ### Message Notification Router _Routes every WhatsApp message to notify, digest, or mute — and reproduces its own results offline._ A multimodal notification router that decides which WhatsApp messages interrupt you, which wait for a digest, and which stay muted, reasoning over text, image posters, and voice notes with per-user history, group dynamics, and sender trust. A safety gate runs first and deliberately blind to engagement history, so a scam can't be argued down by a strong relationship. Evidence retrieval and confidence are deterministic code, not model output, and every model response is cached and committed — the run reproduces byte-for-byte with no API key set. 93% action accuracy on the labelled samples, standard library only. - **Year:** 2026 - **Tags:** Python, Claude, Gemini, Groq, Evals, Data - **Tech:** Python, Claude Haiku, NVIDIA NIM, Gemini OCR, Groq Whisper, Prompt Caching - **Source:** https://github.com/ishanavasthi/personalized-message-router - **Activity:** 82 commits, 2026-07-28 → 2026-08-03 ### Agent-Toolkit Buildability Atlas _An agent pipeline researched 100 apps to answer one question: can we build a toolkit for it?_ An agent pipeline — not a spreadsheet — researched 100 apps across 10 categories on auth method, self-serve access, API surface, and existing MCP support, then graded each on whether an AI-agent toolkit is buildable today. Findings were verified against first-party docs through an accuracy loop, and the whole case study ships as a single self-contained HTML page. - **Year:** 2026 - **Tags:** Python, JavaScript, Multi-Agent, Data - **Tech:** Python, Agent Pipeline, HTML/CSS/JS, Composio - **Source:** https://github.com/ishanavasthi/composio-research - **Activity:** 5 commits, 2026-07-09 → 2026-07-09 ### Swiggy Agent (n8n) _Agentic food-ordering concierge built entirely out of n8n workflows._ An agentic food-ordering concierge with no application code — two chained n8n workflows do the work. A natural-language request like "vegetarian dinner for 2 under 500" is parsed into structured intent, then refined through search, recommendation, and review passes. The restaurant backend is a deliberate mock behind one HTTP node, so swapping in the real API changes exactly one thing. - **Year:** 2026 - **Tags:** n8n, Multi-Agent - **Tech:** n8n, HTTP Workflows, LLM Intent Parsing - **Source:** https://github.com/ishanavasthi/swiggy-n8n - **Activity:** 2 commits, 2026-06-04 → 2026-06-04 ## Projects: RAG & Document AI Retrieval pipelines over documents people actually need to read. ### Lenny Growth Assistant _Grounded answers over 301 Lenny's Podcast episodes, cited to the exact YouTube second._ Conversational assistant over 15,983 passages from Lenny's Podcast: hybrid retrieval with RRF reranking over pgvector answers product and growth questions with citations linking to exact YouTube timestamps. Ships a Ship 30 essay skill, versioned Markdown and HTML artifacts with a sandboxed viewer, and a 9/10 golden eval set. Runs on Anthropic, OpenAI, OpenRouter, or local Ollama with zero keys. - **Year:** 2026 - **Tags:** Python, TypeScript, Claude, RAG, Evals, Full-Stack, Data - **Tech:** Python, FastAPI, Claude Agent SDK, pgvector, LiteLLM, React, Vite, Docker - **Source:** https://github.com/ishanavasthi/lenny-growth-assistant - **Activity:** 65 commits, 2026-08-23 → 2026-08-26 ### ChatWithPDF _NotebookLM-style agentic assistant for querying your own PDFs._ Developed a full-stack, AI-powered document assistant enabling users to upload PDFs and interactively query them using a RAG pipeline with an agentic reasoning loop, inspired by NotebookLM. - **Year:** 2026 - **Tags:** Python, TypeScript, LangGraph, FastAPI, Next.js, Gemini, RAG, Full-Stack - **Tech:** Python, LangChain, LangGraph, ChromaDB, Next.js, Tailwind CSS - **Source:** https://github.com/ishanavasthi/ChatWithPDF - **Live / demo:** https://link.ishanavasthi.in/ChatWithPDF - **Activity:** 11 commits, 2026-05-23 → 2026-05-24 ### LexGuard _Upload a contract, get plain-English risk analysis of every clause._ AI-powered contract intelligence. Upload a PDF or DOCX and get plain-English risk analysis of every clause — red flags, real-world impact, and negotiation tips — instead of a summary that skips the parts that matter. - **Year:** 2026 - **Tags:** Python, TypeScript, FastAPI, Next.js, Gemini, RAG, Full-Stack - **Tech:** FastAPI, Gemini, Next.js 14, Tailwind CSS, PDF/DOCX Parsing - **Source:** https://github.com/ishanavasthi/lexguard - **Activity:** 25 commits, 2026-05-17 → 2026-05-23 ## Projects: RL Environments & Evals OpenEnv-compliant environments for training and grading agents. ### RecruitEnv _OpenEnv RL environment for candidate pipeline triage under a step budget._ An OpenEnv-compliant RL environment simulating candidate pipeline triage: an agent reviews synthetic developer profiles across GitHub, LeetCode, Kaggle, and resume signals and makes shortlisting decisions under a step budget. Built for the OpenEnv Hackathon by Meta x Hugging Face. - **Year:** 2026 - **Tags:** Python, FastAPI, Docker, RL Env, Evals - **Tech:** Python, FastAPI, NumPy, Docker, Pytest - **Source:** https://github.com/ishanavasthi/recruit-env - **Activity:** 33 commits, 2026-04-03 → 2026-05-19 ### InsuranceClaimEnv _OpenEnv environment for high-stakes insurance claim adjudication._ An OpenEnv environment simulating insurance claim adjudication — the document-heavy, high-stakes workflow where an adjudicator reviews a claim against a policy document and must approve, deny, or escalate, with a payout amount and a written justification. - **Year:** 2026 - **Tags:** Python, Docker, RL Env, Evals - **Tech:** Python, OpenEnv, Docker - **Source:** https://github.com/ishanavasthi/insurance-claim-env - **Activity:** 4 commits, 2026-03-27 → 2026-08-14 ## Projects: Developer Tools Small, sharp utilities built to remove a specific daily friction. ### ContextLens _Browser activity capture that survives MV3 service worker death without dropping or duplicating a single event._ A Chrome Manifest V3 extension that captures tab changes, navigation, interactions, and screenshots into Postgres through a backend API. Because MV3 kills the service worker on idle, nothing durable lives in memory: events hit IndexedDB before they count as captured and client-generated ULIDs make retries idempotent. Capture is off by default and granted scope by scope. - **Year:** 2026 - **Tags:** TypeScript, Docker, DevTools, Full-Stack, Systems - **Tech:** TypeScript, Chrome MV3, IndexedDB, Postgres, Docker, Playwright, ULID - **Source:** https://github.com/ishanavasthi/context-lens - **Activity:** 65 commits, 2026-07-31 → 2026-07-31 ### Markdown Copier _Chrome extension that turns any page into clean, LLM-ready Markdown._ Chrome extension that converts a whole webpage or just the selected content into clean, LLM-ready Markdown with one click — no nav chrome, no tracking pixels, no hand-editing before you paste it into a prompt. - **Year:** 2026 - **Tags:** TypeScript, DevTools - **Tech:** TypeScript, Chrome Extension APIs, Turndown - **Source:** https://github.com/ishanavasthi/markdown-copier - **Activity:** 1 commits, 2026-06-20 → 2026-06-20 ### KV-Cache _In-memory key-value store over raw TCP, built on asyncio._ A high-performance in-memory key-value cache server built with Python asyncio, speaking a custom wire protocol over raw TCP sockets. Layered into network, protocol, and cache tiers, with Docker packaging, AWS deployment scripts, and load tests. - **Year:** 2026 - **Tags:** Python, Docker, Systems - **Tech:** Python, asyncio, TCP Sockets, Docker, AWS, Shell - **Source:** https://github.com/ishanavasthi/kv-cache - **Activity:** 1 commits, 2026-01-25 → 2026-01-25 ## Projects: Full-Stack & Web Product-shaped web apps, end to end. ### Verdict Desk _Grades student code in a hardened Docker sandbox, and no AI answer reaches a student until a teacher approves it._ An AI code-grading and doubt-resolution portal built around what happens when the inputs are hostile. Student submissions run in a network-isolated, resource-capped throwaway container per submission — the API deliberately stays on the host rather than mounting the Docker socket into a container. AI drafts answers to student doubts, but the draft → pending → approved state machine is enforced by Postgres triggers, not application code, so an illegal transition fails at the database even if every layer above it is compromised. Ships with an abuse suite that proves containment: fork bombs, infinite loops, network calls, and 100MB of stdout all get contained without moving API memory. - **Year:** 2026 (featured) - **Tags:** TypeScript, Next.js, Docker, Full-Stack, Systems, Evals - **Tech:** Next.js 15, NestJS 11, TypeScript, PostgreSQL, Prisma, Docker, pnpm workspaces - **Source:** https://github.com/ishanavasthi/verdict-desk - **Activity:** 72 commits, 2026-08-03 → 2026-08-05 ### FinTracker _Personal finance tracker with category budgets and live spend breakdowns._ A personal finance tracker for logging income and expenses, visualising where money actually goes, and holding a monthly per-category budget. Supabase auth with protected routes, dashboard summary cards, a category pie chart, a six-month income-vs-expense bar chart, and full transaction CRUD. - **Year:** 2026 - **Tags:** JavaScript, Supabase, Full-Stack, Data - **Tech:** React, Supabase, Supabase Auth, Recharts, Vercel - **Source:** https://github.com/ishanavasthi/fintracker - **Live / demo:** https://fintracker-three-iota.vercel.app - **Activity:** 14 commits, 2026-04-20 → 2026-04-20 ### EventHive _Event discovery and booking platform with a React Native client._ A full-stack event discovery and booking platform: an Express and MongoDB REST API with JWT auth covering users, events, and bookings, paired with a React Native (Expo) mobile client with custom navigation and a glass-card UI kit. - **Year:** 2026 - **Tags:** JavaScript, React Native, Full-Stack - **Tech:** Node.js, Express, MongoDB, JWT, React Native, Expo - **Source:** https://github.com/ishanavasthi/EventHive - **Activity:** 46 commits, 2026-02-20 → 2026-08-29 ### This Portfolio _The site you're reading — Next.js, MDX blog, and this cascade._ Personal site built with Next.js App Router and Tailwind CSS v4: an MDX blog pipeline, a faceted project cascade, and an llms.txt route so agents can read it too. - **Year:** 2023 - **Tags:** TypeScript, Next.js, Full-Stack - **Tech:** Next.js, TypeScript, Tailwind CSS, MDX, shadcn/ui, Framer Motion - **Source:** https://github.com/ishanavasthi/portfolio - **Live / demo:** https://ishanavasthi.in - **Activity:** 58 commits, 2026-04-29 → 2026-08-16 ## Projects: Foundations Earlier work — Spring Boot services, low-level design, Java, and data analysis. ### Product Service _Spring Boot product catalogue microservice with Flyway migrations._ A Spring Boot product catalogue microservice with JPA persistence, Flyway migrations, a FakeStore API fallback implementation behind the same service interface, global exception handling, and Kubernetes deployment manifests. - **Year:** 2024 - **Tags:** Java, Spring Boot, Docker, Systems - **Tech:** Java, Spring Boot, JPA, Flyway, Docker, Kubernetes, JUnit - **Source:** https://github.com/ishanavasthi/Product-Service - **Activity:** 16 commits, 2024-03-31 → 2024-10-10 ### Cart Service _Spring Boot cart service with layered domain design._ A Spring Boot application implementing a shopping cart service, built to practise layered controller/service/repository design and REST API conventions. - **Year:** 2024 - **Tags:** Java, Spring Boot, Systems - **Tech:** Java, Spring Boot, Maven - **Source:** https://github.com/ishanavasthi/cart-service - **Activity:** 10 commits, 2024-03-12 → 2024-03-12 ### Patient Record System _Spring Boot REST API for patients, histories, and appointments._ A Spring Boot REST API for managing patients, their medical histories, and appointments, with a layered controller/service/repository architecture and custom domain exceptions. - **Year:** 2024 - **Tags:** Java, Spring Boot, Systems - **Tech:** Java, Spring Boot, JPA, Maven - **Source:** https://github.com/ishanavasthi/Patient-Record-System - **Activity:** 19 commits, 2024-05-01 → 2024-05-02 ### Book Management _Spring Boot book and author CRUD with a full test suite._ A Spring Boot book and author management app with JPA repositories, JSP views, and test coverage across both the service and repository layers. - **Year:** 2025 - **Tags:** Java, Spring Boot, Full-Stack - **Tech:** Java, Spring Boot, JPA, JSP, JUnit - **Source:** https://github.com/ishanavasthi/spring-book - **Activity:** 3 commits, 2025-05-07 → 2025-05-07 ### Bestsellers Analyser _Java CSV analysis of Amazon's top 50 bestsellers, 2009–2019._ A Java program that reads and analyses Amazon's top 50 bestselling books from 2009 to 2019 out of a CSV dataset, split into model, dataset-reader, and driver layers. - **Year:** 2025 - **Tags:** Java, Systems, Data - **Tech:** Java, CSV Parsing - **Source:** https://github.com/ishanavasthi/lld-assignment - **Activity:** 2 commits, 2025-08-19 → 2025-08-19 ### Image Editor _Image manipulation in pure Java — grayscale, blur, rotate._ Various image manipulation operations implemented in pure Java with javax.imageio: grayscale conversion, brightness adjustment, rotation, and blurring. - **Year:** 2023 - **Tags:** Java, Systems - **Tech:** Java, Java IO, javax.imageio - **Source:** https://github.com/ishanavasthi/image-editor - **Activity:** 8 commits, 2023-11-24 → 2023-11-24 ### Aerofit Sales Analysis _Exploratory case study on Aerofit treadmill sales._ A case study analysing Aerofit sales data across Python's analysis stack — profiling customer segments, comparing product tiers, and visualising the drivers behind each purchase. - **Year:** 2024 - **Tags:** Python, Data - **Tech:** Python, Pandas, NumPy, Matplotlib, Seaborn, Jupyter - **Source:** https://github.com/ishanavasthi/aerofit-analysis - **Activity:** 9 commits, 2024-03-20 → 2024-03-20 ### Easy App _React Native unit converter with live conversion._ A React Native (Expo) unit converter with live conversion for meters/feet and Celsius/Fahrenheit, numeric input validation, and a clean responsive UI. - **Year:** 2025 - **Tags:** JavaScript, React Native, Full-Stack - **Tech:** React Native, Expo, JavaScript - **Source:** https://github.com/ishanavasthi/rn-assignment-ishanavasthi - **Activity:** 1 commits, 2025-11-26 → 2025-11-26 ### Overloading _Unity game-jam prototype about not tipping over a loaded truck._ A 3D Unity truck-simulator game-jam prototype with one focused mechanic: drive a top-heavy overloaded truck to the finish line without tipping. A raised centre of mass, lateral tip forces, and steering roll torque make aggressive driving lose. - **Year:** 2026 - **Tags:** C# - **Tech:** C#, Unity 6, WebGL - **Source:** https://github.com/ishanavasthi/OverloadingMVP - **Activity:** 3 commits, 2026-04-20 → 2026-04-29 ## Writing ### Your agent got 3x more expensive and every test still passed - **URL:** https://ishanavasthi.in/blog/catching-agent-regressions-in-ci - **Published:** 2026-07-27 - **Reading time:** 22 min - **Summary:** Building a CI gate for AI agents, with SigNoz as the datastore rather than a dashboard on the side. Every bug that mattered was silent, and five separate green lights meant nothing. Here is a pull request. It changes nine lines of a system prompt — something about being more thorough before answering. The unit tests pass. Coverage is unchanged. Lint is clean. You would approve it. It also makes the agent cost **150% more per task**, take **twice as many retrieval hops**, and start calling a tool it has never touched before. None of that is in the diff, and none of it is in your CI. The code did not change shape — same functions, same signatures, same assertions. What changed is *behaviour*, and behaviour is not something a test suite has an opinion about unless you build one. You find out in next month's invoice. Or in a latency alert at 2am, twenty pull requests later, when nobody remembers which one did it. So I built the thing that catches it. What follows is the engineering, and the through-line turned out to be sharper than the product: **almost every bug in this project was silent, and five separate times a green light meant nothing.** Building a tool to catch invisible regressions is an unusually good way to discover how much of your own feedback is fake. #### What it does Preflight is a CI check for agent behaviour. On every pull request it runs a six-case golden suite against both the merge base and the PR head, compares six metrics, and blocks the PR if any of them regressed past a committed threshold. ``` pull request opened | +-- foundryctl cast ......... SigNoz, stood up inside the CI job +-- git merge-base .......... resolve the baseline SHA | | suite @ merge-base suite @ PR head | 6 cases 6 cases | \ / | \ OTLP/HTTP :4318 / | +------------+------------+ | | | v | +--------------+ | | SigNoz | | +------+-------+ | | POST /api/v5/query_range | v | +--------------+ +-------------------->| the gate | +------+-------+ | exit 0 clean ............ | ..... sticky PR comment, green exit 1 metric breached .. | ..... sticky PR comment, red, blocked exit 2 ingest timed out . | ..... "the gate could not run" exit 3 nothing to compare | ..... "the gate could not run" ``` Exit 1 and exit 3 being different codes is not fussiness. A gate that cannot run must never render as a gate that failed — an expired baseline showing up in CI as "your agent regressed" burns a team's trust in the check faster than any false negative. Every path that produces no report synthesises one, because silence in CI is exactly how a broken gate goes unnoticed, which is the failure mode this whole thing exists to prevent. The comment it posts: ``` baseline candidate delta threshold FAIL Cost / task $0.0026 $0.0064 +150.5% +25% FAIL Tokens / task 2,034 4,670 +129.7% +25% FAIL p95 latency 3.18s 8.51s +167.4% +75% FAIL Tool calls / task 1.33 3 +125.0% +40% FAIL Retrieval hops 1.00 2.33 +133.3% +50% ok Success rate 100% 100% - drop > 1pt Biggest mover: damaged-item moved $0.0070 (+5,100 tokens, +3 tool calls) ``` Every row deep-links to that case's trace waterfall. #### The bet: SigNoz is the datastore, not a dashboard on the side The obvious design is: run the agent, write results to a JSON file, compare against a stored baseline, fail the build. That works, and it is boring, and it throws away the most interesting property of the problem. Agent runs are already *traces*. A task is a root span, each model call is a child, each tool call is a child of that. If you are going to instrument the agent anyway — and you are, because you will want to debug it — then the trace store already contains everything the gate needs. Writing a second, parallel record into a JSON file is duplication that will drift, and the day the dashboard and the PR comment disagree in front of a reviewer is the day the gate is finished. So: **there is no local results file.** Every number in that comment came back out of `POST /api/v5/query_range`. If SigNoz cannot answer, the gate fails loudly rather than guessing. That rested on one question I could not answer from the docs: *can the query API aggregate and group by a custom span attribute?* Not a resource attribute, not a well-known field — an arbitrary `eval.case_id` I invented. If the answer was no, the design was dead and I needed to know in the first hour, not the fifth. It was the first thing I built, ahead of anything that looked like a feature. It is yes: ```jsonc "aggregations": [{ "expression": "sum(preflight.cost_usd)" }], "filter": { "expression": "attribute.vcs.commit_sha = '59607e52'" }, "groupBy": [{ "name": "eval.case_id", "fieldContext": "attribute" }] ``` One row per case, per commit, in a single round trip. Everything else follows from that being true. #### What the data looks like Spans are hand-rolled against the OpenTelemetry GenAI semantic conventions rather than emitted by an auto-instrumentation library. That is more code, on purpose: the gate reads spans back *by attribute name*, so the names have to be exactly the spec's rather than whatever a library decided to call them. Every `gen_ai.*` attribute in use is at Development stability in the spec, and I said so in the README rather than hoping nobody checked. One case, as a trace: ``` eval.case damaged-item 8530 ms ├─ retrieve policy-kb (grounding hop) ├─ chat claude-haiku-4-5 1425 ms │ gen_ai.usage.input_tokens = 1904 │ preflight.cost_usd = 0.00121 ├─ execute_tool policy_search │ └─ retrieve policy-kb ├─ chat claude-haiku-4-5 1088 ms ├─ execute_tool lookup_order ├─ chat claude-haiku-4-5 1116 ms ├─ execute_tool check_inventory ├─ chat claude-haiku-4-5 1278 ms ├─ execute_tool policy_search │ └─ retrieve policy-kb └─ chat claude-haiku-4-5 3622 ms ``` Cost is computed inside the LLM span on exit, from a price table committed alongside the thresholds, so `preflight.cost_usd` and the token attributes can never disagree with each other. `policy_search` nests a retrieval span, so reaching for it costs a hop — which is what makes "retrieval hops per task" a real behavioural metric rather than a proxy. The whole system is about six moving parts: ``` agent/reference.py ......... tool-calling loop, claude-haiku-4-5 preflight/instrument.py .... case / llm / tool / retrieval spans preflight/otel.py .......... OTLP exporters: traces, metrics AND logs | | :4318 v S i g N o z (deployed by Foundry from a committed casting.yaml) | +-- /api/v5/query_range ---> preflight/query.py | | | contracts.RunSummary | | | preflight/differ.py ---> exit code | | | preflight/report.py ---> PR comment | +-- MCP server :8000 ------> scripts/signoz_apply.py (dashboards, alerts) preflight/diagnose.py (the explainer) ``` The metric arithmetic lives in exactly one place — a `RunSummary.metric()` on the shared contract, not in the differ. The gate and the renderer cannot disagree about what "cost per task" means, because there is one implementation of it. #### The regression, seen twice The seeded regression is a prompt edit. Not a code change — the agent's *instructions*, in English: ```diff - "Use the tools to check facts before answering. - Answer in at most two sentences." + "Work strictly one step at a time and call exactly one tool per step. + First call policy_search for the general rule. Then call lookup_order + for the order. Then call check_inventory for the SKU you found on that + order. Then call policy_search once more for the specific rule that + applies to that item's status. Only after all four steps, enumerate + every option available to the customer with its trade-offs." ``` And here is the same test case before and after, as SigNoz renders it: ``` baseline e0592cf8 candidate 59607e52 5 spans, 2.18 s 13 spans, 8.53 s eval.case damaged-item eval.case damaged-item ├─ retrieve policy-kb ├─ retrieve policy-kb ├─ chat ├─ chat ├─ execute_tool lookup_order ├─ execute_tool policy_search <- new └─ chat ├─ retrieve policy-kb <- new ├─ chat ├─ execute_tool lookup_order ├─ chat ├─ execute_tool check_inventory <- new ├─ chat ├─ execute_tool policy_search <- new ├─ retrieve policy-kb <- new └─ chat ``` Read the prompt, then read the trace. `policy_search`, `lookup_order`, `check_inventory`, `policy_search` again — the waterfall is literally the four steps the instructions asked for, in order. Cause in the diff, effect in the trace, and the connection needs no explaining. #### What actually inflates agent cost This is the most portable thing I learned building it, and it took three failed attempts to see. My first two seeded regressions asked the model to be more thorough. "Enumerate every option." "Check every relevant tool, one per step." Both landed at **+16 to +18% tokens** — squarely in the "15%, not 3x" range that makes for a demo nobody believes. The obvious read was that the prompt edit was too weak. It wasn't. The model was **batching its tool calls into a single parallel turn**. More tools, same number of turns, input context re-sent the same number of times. Verbosity is cheap. What actually moved the number was a **dependency chain**: call `policy_search`, then `lookup_order`, then `check_inventory` *for the SKU you found on that order*. Each tool's input requires the previous tool's output, which forces genuinely sequential turns — and each turn re-sends a growing transcript. Result: +130% tokens, +133% retrieval hops, +150% cost. > **Turn count is the multiplier. Output length is rounding error.** If you are > reasoning about what an agent costs, count the round trips, not the words. There is a second half to this, and it nearly cost me the finding. Before I saw the batching, I had set `MAX_MODEL_TURNS = 3`. The cap was truncating the trajectory, clipping a longer-running agent back into looking like the baseline. Raising the ceiling to 5 changed nothing for the baseline — every case still terminated on `end_turn` in two calls, every cassette still hit — and let the regression show its real size. **Check the instrument before concluding the effect is small.** #### Reading the query API Four things about `/api/v5/query_range` that cost me time and are not obvious from a first read of the docs. **Aggregation aliases are not echoed back.** You send an `alias`, and the response ignores it. Columns come back as `__result_0`, `__result_1`, … in the order the aggregations were requested. Group-by columns are named after the attribute and marked `columnType: "group"`. Rows are nested at `data.data.results[].data` as *positional lists* — not under an `aggregations[]` key, which is what I had assumed and built a parser against. The flattener has to re-attach aliases by aggregation index. **Traces and metrics have entirely different response shapes.** Traces come back as `results[].columns[]` plus positional `results[].data[]`. Metrics come back as `results[].aggregations[].series[].values[]`. A helper written for one silently returns an empty list when pointed at the other — which downstream reads as *no data*, which reads as *no regression*. **An unqualified attribute name resolves to the resource attribute, not the span attribute.** `vcs.commit_sha = 'abc'` matched zero spans while `attribute.vcs.commit_sha = 'abc'` matched forty-two, because the resource is stamped once per process. In ordinary CI — one process, one commit — the two values agree and the bug is invisible. It only surfaces when they diverge. `groupBy` is immune because it demands an explicit `fieldContext`, which is exactly why my first probe looked fine and gave me false confidence. Qualify every filter. **`max(timestamp)` works and returns epoch seconds; string intrinsics cannot be aggregated at all.** That first fact is load-bearing: resolving a commit SHA to its most recent run is one scalar query grouped by run id, aggregating `count()` and `max(timestamp)` — no bisection over time windows. But `any()` is not a recognised function, and `max(trace_id)` makes ClickHouse try to cast hex to Float64 and fail. Trace ids have to come out through a `fieldContext: "span"` group-by instead. #### A query that scanned 48,527 rows and returned none The run-level metrics were the hardest read path in the project, and the shape of the failure is worth the space. Six OpenTelemetry histograms are emitted per case on the same dimensions as the spans. Every `signal: "metrics"` query I tried returned zero rows — the base name, `.sum`, `.bucket`, with and without cumulative temporality, across several space-aggregation values — while the data was plainly visible in ClickHouse. Two separate things were going on. **First, a histogram does not exist under its own name.** SigNoz asks about the metric and gets back a message as unambiguous as error messages get: ``` metric preflight.case.cost_usd has never been received. Check the metric name and instrumentation ``` The instrumentation was fine. On ingest a histogram is decomposed into `preflight.case.cost_usd.sum`, `.count`, `.bucket`, `.min`, `.max` — and the base name exists as *nothing at all*. That warning reads as a verdict about your exporter, and the obvious response is to go rewrite working code. It is actually a statement about a name. **Second, and the real blocker: letting the backend auto-detect the metric type routes you down a dead end.** SigNoz catalogues these as `Histogram / Cumulative / isMonotonic`, and a query that auto-fetches that metadata takes a histogram-percentile path which returns zero rows *while still scanning 48,527 of them*. The scan count is the tell. Pinning every inferred parameter explicitly is what fixes it: ``` metricName: preflight.case.cost_usd.sum # the suffixed series, not the base metricType: gauge # explicit; do NOT let it auto-fetch timeAggregation: max # explicit spaceAggregation: sum ``` Cross-checked against the trace-derived total for the same commit: **$0.0769 both ways**, exactly. > `rowsScanned > 0` with an empty result set means the filter matched and the > *aggregation* threw everything away. That is a completely different bug from > `rowsScanned == 0`, and the two are indistinguishable if you only look at the > row count. And then the dashboards read traces anyway, which is worth being honest about. Cracking the metrics path did not change the design: the gate reads traces, so a trace-backed dashboard cannot disagree with the PR comment, while a metrics-backed one can. Trace aggregation also has no temporality or bucket-alignment subtleties. Solving a blocker and then deliberately not depending on it is a legitimate outcome. #### Every bug that mattered was silent Not one of the real bugs threw an exception. Every one of them returned a plausible number. **A percentage of a near-zero number is not information.** The first end-to-end run reported a +205% latency regression between two *identical* runs, because the cases took 0.12ms. A gate that cries wolf gets switched off by the third engineer who hits it. Latency now has to clear an absolute floor as well as a percentage, and it is the only metric with a near-zero regime, so it is the only one with a floor. That is also why the thresholds are measured rather than guessed. Running the real suite twice under two different SHAs and diffing it gives **0.0% on cost, tokens, tool calls and hops** — replay makes those exact — and **0.2% on latency**. The thresholds sit at 25–75%, which is over 100x the measured noise, and they are sized for *re-recorded* cassettes rather than for that 0%. **A reader's default lookback window silently truncated valid diffs.** The summary reader defaulted to sixty minutes; the SHA resolver would happily find a baseline run from yesterday. The result was not an error, it was "baseline run has no cases" — which reads like a completely different problem. **A field that looks like an idempotency key, and isn't.** SigNoz dashboards carry a top-level `name`. Posting the identical payload twice creates *two* dashboards with the same name and different UUIDs. So the apply script reconciles rather than upserts: list, match on name, update in place by id, delete stale duplicates. Inside that one, a smaller and nastier version: `signoz_list_alert_rules` returns the rule UUID as `ruleId`, while `signoz_create_alert` returns it as `id` and `signoz_update_alert` expects `id`. Reading `id` off a listing yields nothing; the update then targets nothing and **still reports success**. I only caught it because a debug print showed a null id next to a rule that was demonstrably firing. **A test double that is faithful in the wrong dimension.** The cassette cache replayed tokens and content perfectly and dropped latency on the floor — replayed cases finished in ~0.2ms. Tokens and cost were exact, so the cache looked correct, and `p95_latency_ms` had quietly become a random number generator: ordinary sub-millisecond jitter is a +150% swing against any threshold anyone would pick. That is the "gate is flaky, so the team switches it off" failure arriving *through the component built to make the gate stable*. The fix is to record the provider's wall time in the cassette and sleep it on replay. Offline fixture tests caught none of these. A single run against real SigNoz caught all of them. Fixtures test the code you wrote; these were all bugs in the code I *thought* I had written. #### Five green lights that meant nothing SigNoz is a single-page app behind a catch-all route. **Every** unmatched path returns HTTP 200 with the same HTML shell. That one fact manufactured a false positive five separate times, in five different disguises. 1. **The auth API.** The documented `/api/v1/login` and `/api/v1/pats` no longer exist. They fall through to the catch-all, so a client that checks the status code sees success and then dies on JSON parse. The real surface is `POST /api/v2/sessions/email_password`, which needs an org id. A related trap one layer down: service-account keys start with **zero** permissions, so a fresh key authenticates perfectly and then returns `authz_forbidden` on every call until a role is attached. 2. **The trace deep links.** `/trace/deadbeef` and `/trace/this-is-not-a-trace` both return 200. A status-code check would have "verified" a URL format that was entirely wrong. The real check is the request the page makes to render itself — `POST /api/v4/traces/:id/waterfall`, asserting a non-empty span list, which returns `"type": "not-found"` for a bogus id. 3. **The dashboard panels.** A panel with a subtly wrong filter renders perfectly and is simply empty, indistinguishable from a healthy panel on a quiet system. So the verifier re-executes every committed panel query — **using each panel's own declared request type**, because scalar and time-series return different shapes and counting the wrong one reads as zero. Legitimately-empty panels go in an allowlist *with a written reason*, otherwise "expected empty" quietly becomes "we stopped looking." 4. **The dashboards themselves.** Opening a committed dashboard in the UI showed the brand-new-dashboard empty state — blank title, no panels. Every API check said otherwise: four dashboards present, six panels each, 26 of 27 panel queries returning rows. The data was never the problem. The dashboards use SigNoz's v6 schema, which only the newer frontend renders, and that sits behind a feature flag which ships off. The flag has no write API (the writes returned the catch-all HTML, of course) and is set through config, where underscores inside a flag name are *doubled* in the env-var form — not something you guess. I found it by grepping the server binary for `SIGNOZ_[A-Z_]*` and spotting a shipped sibling flag. 5. **Trace existence.** `/api/v1/traces/:id` returns 200 for traces that do not exist. I used it to confirm some traces had survived a restart, got a green light that meant nothing, and only caught it by querying ClickHouse directly. > **When a system has a catch-all route, status codes stop carrying information > for every endpoint — not just the ones that burned you.** Any check a fallback > can satisfy has to assert on content: a parsed field, a row count, a span list. > Otherwise it is not a check. There is a companion lesson about *where to look*. Four times, the fastest path to the truth was the shipped frontend bundle. The deep-link format is the clearest example — rather than copying a URL out of the address bar, the bundle gave four independent confirmations: the router constant `ROUTES.TRACE_DETAIL: '/trace/:id'`; the trace-detail chunk reading `searchParams.get('spanId')` to decide which span to expand; the UI's own **Copy link** handler building exactly that string; and the waterfall API accepting the id. When vendor docs and a running instance disagree, the client is a complete, current, executable description of the API, and it gives you the format *and* the reason. #### The failures that only appear in CI The gate worked locally, first try, every time. Then I opened a real pull request and it failed with `0/32 spans`. Not a partial count. Zero. Meanwhile every health check was green — SigNoz's `/api/v1/health` returning 200 the whole way through. The collector logs had it. About thirty-five seconds into the run: ``` "msg":"Shutdown complete." "msg":"Starting collector service" ``` SigNoz's collector fetches its configuration over opamp shortly after boot and **restarts**. Anything exported during that window is accepted over OTLP — returns 200, no error, no warning — and then dropped. Locally this never bit me because SigNoz had been running for an hour before I ever ran the suite. It only happens on a cold deployment, which is precisely what CI is. The fix is not a longer timeout. It is gating on the property you actually depend on: write a probe span, then poll the query API until it reads back. Health means "the API process is up." It does not mean "the pipeline persists data," and those are different claims. The general version already existed elsewhere in the runner: after flushing, it waits until the run's spans are queryable and **fails loudly** rather than diffing a partial run. That guard earned its keep before it ever reached CI — its first act was to report `0/6` and catch a bug in my own response parser, which would otherwise have returned an empty list that downstream code would have read as *no regression*. A component that fails loudly on the expected failure mode also surfaces the unexpected ones. There is a smaller sibling worth knowing about, one span's resolution down. The first run of the diagnosis check failed reporting the root span missing — it was there 200ms later. **The root span closes and exports last**, so "the trace exists" goes true before the trace is *complete*, and a poll that breaks on the first non-empty read observes a torn trace. Wait on the last-written span, not on any span. And one that is pure CI plumbing but cost an hour: > **GitHub Actions runs steps under `bash -eo pipefail` by default, so > `code=${PIPESTATUS[0]}` never executes.** The step dies on the failing command > before it can capture the exit code it exists to capture — and the gate's whole > design is "post the comment, *then* fail." Every step that inspects an exit code > has to `set +e` first. I found that by extracting each `run:` block out of the workflow YAML and executing it under `bash --noprofile --norc -eo pipefail` locally, which is worth doing for any workflow you cannot afford to debug by pushing commits. #### Dashboards as code, and four shapes the schema does not describe Four dashboards and two alert rules are committed as JSON and reconciled onto SigNoz through its **MCP server** rather than the REST API. The committed JSON *is* the MCP tool argument, so there is no payload translation layer to drift out of sync. The MCP server publishes an `inputSchema` per tool, and it is a faithful rendering of the underlying Go types. It still leaves four things to be discovered by HTTP 400: 1. **`schemaVersion` must be the string `"v6"`** — typed as a bare string with no enum, so nothing tells you the set of legal values. 2. **The grid is 12 columns wide in total, not 24.** A half-width panel is `width: 6`, which looks like a quarter until you find out. 3. **Table panels format per column** via a `columnUnits` map, and reject a bare `unit` with *"json: unknown field"*. Every other panel type takes `unit`. 4. **A panel accepts exactly one query entry.** Multi-query arithmetic goes *inside* a single composite-query plugin carrying the input builder-query specs plus a formula — not three sibling entries, which fails with *"panel must have one query"*. Probing for those took four cycles and no amount of reading would have shortened it. Which is an argument for treating a published schema as a description of *types*, not of *validity*. #### Making an agent explain itself When the gate fails, a second agent investigates. It reads the traces over the **SigNoz MCP server** — that is its only source of facts, it cannot read the repository — and explains what happened in English: > The **damaged-item case** is the worst regressor: cost increased from $0.00246 > to $0.00946 (+284%), tokens from 2,016 to 7,116 (+253%). In the baseline it > made 1 tool call and 1 retrieval; in the candidate it makes 3 tool calls > (`lookup_order`, `check_inventory`, and `policy_search` called twice) and 3 > retrievals. It is itself instrumented, so its investigation lands in SigNoz as a trace of its own — one span per reasoning turn, one per query, a couple of dozen in total. One failed gate produces the agent's traces, the gate's queries against them, and the diagnosis agent's own reasoning, all in one place. Writing the acceptance check for that taught me the most useful thing here. My first instinct was "assert the explanation names the worst case." That check is nearly worthless: the gate's per-case table is *in the agent's prompt*. A model that called nothing and read nothing could pass it by paraphrasing its own input. What makes the check real is that `policy_search` and `policy-kb` appear nowhere in the prompt. They exist only as span attributes. Quoting them is proof the agent went to SigNoz and came back. > **When you test a generated explanation, find a fact that lives only on the far > side of the tool call, and assert on that. Everything else measures fluency.** That check immediately earned its keep. I A/B'd a tightened prompt that demanded the model name a concrete root cause and rule out alternatives. It reasoned *visibly better* — correctly identified a prompt edit, explicitly excluded a model swap because every span still showed the same model. It also stopped grouping by tool name, and therefore asserted "the same tools appear in both runs," which is flatly false. The check failed it. Same model, same data, same tools: instructing it harder about *what to conclude* pulled effort away from *gathering what the conclusion rests on*. I kept the looser prompt and recorded the A/B next to it so nobody "improves" it back. Two engineering notes from that piece which generalise well beyond it. **Compaction is a cost control, not a nicety.** A trace-details response is 17.5k characters, of which the envelope, per-column metadata, and the roughly thirty always-null well-known fields every span row carries are almost all of it. Compacting gets it to 5.1k. That matters more than it looks, because in a tool-use loop the whole transcript is re-sent every turn — one fat tool result is not paid once, it is paid on every subsequent turn. The same logic applies to the schemas: the server advertises 42 tools, and pasting their faithful, verbose `inputSchema`s would cost over 10k input tokens *per turn*. The client reads them live but keeps 2 tools and only the parameters the agent is allowed to set, so a schema that changes upstream is a loud failure at startup rather than a 400 mid-investigation. **The diagnosis nearly poisoned the gate.** The instinct is to stamp diagnosis spans with the SHA they are about. That would have been an outage in waiting: the differ resolves a SHA to its *most recent* run, so a diagnosis emitted after the candidate suite wins that race, and the next gate run diffs the agent against its own explanation. Diagnosis spans therefore carry a deliberately distinct `vcs.commit_sha` under their own service name. Same family as the dashboard `name` trap — a field that looks like a correlation key is a *selector* somewhere else. #### Determinism, and where it ends Every model response is recorded to a cassette keyed on a hash of the request — model, system prompt, messages, tools, max tokens, temperature — and the cassettes are committed. CI runs with replay forced on, so the gate makes **zero** model API calls in GitHub Actions, and a fresh clone runs the entire suite offline and free. Two problems, one fix: a gate whose numbers move between runs is not a gate, and paying per PR for a check that runs on every PR is not a check anyone keeps. Total model spend across the whole project was $0.22, gated call by call through a file-locked ledger that fails closed. A detail that is easy to get backwards: the suite runs at `temperature=0`, but *not* for determinism at inference — it does not give you that. It is so that re-recording cassettes does not rewrite every expected answer and force the assertions to be re-tuned. It is worth being precise about where determinism ends. The agent's cassettes key off prompts that never change, so they always hit. The *diagnosis* agent's transcript embeds live SigNoz results, so its cassette only hits while SigNoz still holds the data it was recorded against — which, on someone else's stack with their own runs, it usually will not. That is the general shape, and it is the same lesson as the aggregation aliases: **anything that varies between runs and reaches the request is part of your cache key, whether you meant it to be or not.** Rounding the query window up to a ten-minute bucket removed the obvious source of drift. It did not remove the tool results themselves. So the honest position is a best-effort replay that fails loudly with a self-describing error, plus a live path that costs about five cents — rather than claiming a reproducibility I do not have. #### What I'd do differently **I'd spend the first hour on interfaces, not the first feature.** One dataclass for what the query layer returns, one for what the gate produces, both written before anything consumed them. That is what let three parts of the system be built against a stable seam instead of a moving target, and it is why the metric arithmetic has exactly one implementation. **I'd distrust confident error messages sooner.** "Has never been received — check the metric name and instrumentation" was a statement about a *name*, and I nearly deleted a true claim about working code because of it. Verification has to run in both directions: not only "is this claim too strong?" but "is this failure real?" **I'd write the "does it render?" check at the same time as the "does the API return it?" check.** Four dashboards were present and queryable and completely invisible for hours. The claim was never "the API returns four dashboards" — it was "someone opens this and sees panels", and I verified the wrong one. Anything served through a UI needs at least one check that follows the same path the UI does. **I would not change the bet.** No results file, everything through the query API. It made the gate harder to build and it is the reason a reviewer can go from a number in a PR comment to the exact span that explains it in one click. --- *Source: [github.com/ishanavasthi/preflight](https://github.com/ishanavasthi/preflight)*