I’ve spent the last decade watching the industry swing from "simple regression models" to "deep learning" to the current "agentic" gold rush. Here is what I’ve learned: marketing teams love demos where agents appear to be autonomous savants solving complex problems in seconds. Engineering teams, however, know that those demos are built on perfect seeds, cached API responses, and clean-room inputs.
When we move these workflows into Get more information production—handling real-world customer data, interacting with brittle third-party APIs, and dealing with the inevitable 2 a.m. service outage—the "agent" illusion crumbles. If you aren't red teaming your agents as if they were distributed systems, you aren't building a product; you’re building a liability.
Before you ship, you need a checklist. Not a hand-wavy marketing checklist, but a cold, hard engineering rubric for what happens when your orchestration layer meets the chaotic reality of production.
The Red Team Checklist: Production vs. Demo Gap
Before we dive into the vectors, let’s frame the difference between a "demo-only trick" and a production-grade agent.
Metric Demo Reality Production Reality Seed/Determinism Fixed seed, zero temperature Stochastic behavior, varying token usage API Reliability Cached outputs 5xx errors, rate limits, 2 a.m. timeouts Context Window Short, curated logs Bloated history, noise-to-signal drift Orchestration Happy path execution Deadlock, circular reasoning, retry hell1. The Infinite Tool-Call Loop (The $10k Morning)
The most common failure mode in multi-agent orchestration is the "infinite recursion" trap. An agent hits a tool, receives a confusing output, decides to call the tool again to "correct" it, receives the same output, and enters a state-loop. Without a hard, stateful limit on tool-call depth or cost-per-turn, your agent can drain your budget in minutes.
Red Team Action: Inject adversarial input designed to provoke "repair" loops. Force the agent to interact with a mocked endpoint that returns identical error messages to see if your orchestration layer has a hard-wired circuit breaker. If your agent doesn't have a "give up" threshold, you aren't ready for production.
2. Context Poisoning & Prompt Injection
Marketing folks love to call these "jailbreaks." Engineers should call them "unintended state mutations." An agent reading an email or a database record is effectively executing untrusted code. If your agent is allowed to read user-generated content and then feed that into a function call, you are susceptible to indirect prompt injection.
Red Team Action: Test for "Command Hijacking." Place invisible tokens or specific instructions in a document the agent is supposed to "summarize." If the agent interprets those tokens as instructions to perform a tool call (e.g., "Transfer balance to X"), your orchestration layer lacks sufficient sandbox permissions. Agents should follow the principle of least privilege—not just in auth, but in instruction flow.
3. Orchestration State Corruption
In a multi-agent system, agents often hand off tasks to sub-agents. What happens when Agent A finishes, but the state passed to Agent B is corrupted, incomplete, or partially mutated? In production, memory is not a shared object; it’s a serialized string being passed over a network. Race conditions are real.
Red Team Action: Use "Chaos Engineering" for your orchestration layer. Kill the sub-process between two agent calls. Does the parent agent hang indefinitely? Does it retry a state-mutating tool call twice because it didn't receive an acknowledgment? If your agent doesn't handle partial failures gracefully, it will leave your database in a zombie state.
4. Latency Cascades and Performance Budgets
An agent making three serial tool calls with a 2-second LLM inference time per turn is already pushing an 8-second latency budget. If the orchestration layer adds a retries-on-failure policy, that latency can spike to 30+ seconds. In a customer-facing call center or web UI, that is a production incident.
Red Team Action: Simulate network jitter and increased model latency. Use a tool to introduce artificial 500ms latency on every external API call. Does the agent's logic degrade as the latency increases? Does the user interface time out? You need a "Time-to-First-Token" budget that accounts for the cumulative overhead of your agentic chain.
5. Non-Deterministic API Failures (The 2 a.m. Scenario)
This is where I stop trusting "LLM magic." What happens when your underlying vector database or CRM API flakes at 2 a.m.? Most agents are programmed to be "helpful," which means they might hallucinate a successful outcome if the API returns a null or a partial payload. A "helpful" agent that reports "Task Complete" when it actually failed to write the data is the worst-case scenario for system integrity.
Red Team Action: Perform "negative testing." Shut off the downstream services one by one during a live agent session. Does the agent correctly identify the failure, or does it guess? If the agent interprets a 404 as "data not found, so I'll create it," you’ve introduced a logic bug into your production data pipeline.
6. Unauthorized Escalation and Tool Misuse
Agents are often granted "God mode" tools (like SQL execution or email sending) for the sake of demo-convenience. This is the ultimate security testing failure. An agent shouldn't have access to your whole toolset; it should have access to a scoped, audited, and strictly permissioned set of functions.
Red Team Action: Attempt to "Social Engineer" the agent. Ask the agent to use a tool that is supposedly restricted for that user context. If the agent finds a way to use a sensitive tool to fulfill a "helpful" request, you have failed the most basic security audit. Always build a validation layer between the agent's intent (the tool call) and the agent orchestration production actual execution.
The Engineering Takeaway
I am tired of seeing companies deploy "orchestrated chatbots" and calling them agents. Real agents are distributed systems. They require:
- Idempotency: Every tool call must be safe to retry. Observability: You need a trace of the "thought process" for every single step, not just the final output. Strict Schema Enforcement: If your agent generates JSON, you need a Pydantic-style validator between it and your production API. No exceptions. Human-in-the-Loop (HITL) Gateways: For any destructive or external-facing action, require a hard override if the confidence score drops below a certain threshold.
My advice? Write the failure checklists before you write the prompt engineering logic. If you can't describe exactly how your agent will fail at 2 a.m. when the production database is undergoing maintenance, you aren't ready to push to production. Stop playing with demos and start building for the 99.9% uptime requirement.