Agent Architecture: Empirical Research Edition

The evaluation problem in agent systems is significantly harder than in static NLP benchmarks, and most teams underestimate this by an order of magnitude. A static model evaluation asks: given input X, does the model produce output Y? An agent evaluation asks: given environment E and goal G, does the agent achieve G over a trajectory of N steps, using tools T, while satisfying constraints C? The state space explodes combinatorially.

Three failure modes dominate production evaluation programs:

Evaluating the demo, not the distribution. Teams build evaluation sets from their best-case examples — clear prompts, cooperative environments, well-specified goals. Production traffic is messier: ambiguous requests, edge cases, adversarial inputs, compounding errors. An agent that scores 94% on a curated benchmark and 71% on production traffic is not a rare exception. It is the norm.

Treating LLM-as-judge as ground truth without calibration. Using a capable model (GPT-4o, Claude Sonnet) to evaluate agent outputs is a valid and scalable methodology. The problem is that uncalibrated judge models have systematic biases: they favor longer responses, responses that sound confident, and responses that match their own stylistic patterns. Without a calibration step against human judgments on a representative sample, your eval pipeline has an unknown and potentially large systematic error.

Ignoring trajectory evaluation in favor of output evaluation. If your agent uses 14 tool calls to accomplish a task that should require 3, and produces the correct final output, most output-only evaluation systems will score it as a success. In production, that 14-call trajectory means higher latency, higher cost, higher error probability, and worse user experience. Trajectory efficiency is a first-class metric.

These 12 items represent the failure modes that teams consistently discover in production rather than staging. Work through this list before declaring any agent system production-ready.

1. **Idempotency verification** — Every irreversible action (send, create, charge, post) has been tested for duplicate execution. What happens if the agent runs the same action twice?

2. **Rate limit handling** — All external API calls have retry logic with exponential backoff. The agent degrades gracefully when rate-limited rather than looping.

3. **Context window exhaustion test** — What happens in session 50, after the context is full? Has this been tested explicitly?

4. **Adversarial prompt test** — Has the system been tested against prompt injection via user input, retrieved documents, and tool outputs?

5. **Tool failure cascade test** — What happens when a tool the agent depends on returns an error? Does the agent recover gracefully or spin?

6. **Human escalation path** — Is there a defined and tested path for the agent to escalate to a human when it detects it is operating outside its competence boundary?

7. **Audit log completeness** — Every agent action is logged with enough context to reconstruct the decision. Logs are stored outside the agent's own memory.

8. **Cost budget enforcement** — There is a hard ceiling on token spend and tool call count per session, enforced at the infrastructure level, not the prompt level.

9. **PII handling verification** — Any personally identifiable information that enters the agent's context has a documented handling policy and is not logged in plaintext.

10. **Rollback procedure** — There is a documented and tested procedure to reverse any action the agent can take that has real-world consequences.

11. **Model version pinning** — The production deployment is pinned to a specific model version. Automatic model updates are disabled.

12. **Evaluation pipeline coverage** — The automated eval pipeline covers at least 80% of the task categories present in production traffic.