Five questions. Pick the best answer for each. Scoring at the bottom.
1. Your team wants to deploy an agent to handle inbound customer-service tickets. The most important early investment is:
a) The largest foundation model your budget allows
b) A clearly scoped permissions model — what the agent can read, write, and send
c) A bigger context window so the agent can see more ticket history
d) A custom fine-tune on your historical tickets
Correct Answer: b) Permissions are the design surface. Model size and context window matter less than what the agent is allowed to touch when it makes a mistake.
2. A vendor pitches their product as “fully agentic.” The strongest screening question is:
a) “What foundation model do you use?”
b) “If we remove the next step from the prompt, does the system still complete the task?”
c) “Is your context window at least 200K tokens?”
d) “Do you support fine-tuning?”
Correct Answer: b) If removing the next step breaks the system, it’s an automated workflow with AI components — not an agent. Real agency means the system can plan its own next step.
3. Your agent has 95% per-step accuracy across an eight-step process. The end-to-end success rate is approximately:
a) 95%
b) 88%
c) 66%
d) 40%
Correct Answer: c) 0.95⁸ ≈ 0.66. Per-step accuracy compounds. Pick problems with short chains or strong recovery.
4. Which is the strongest predictor that an agent deployment will fail in production?
a) The foundation model is from a competitor
b) The team can’t replay the agent’s plan, tools called, and inputs/outputs
c) The agent uses a smaller context window than the latest frontier model
d) The cost per task is higher than the human baseline
Correct Answer: b) Without observability you can’t diagnose failure, can’t tune the loop, and can’t pass an audit. The most common cause of stalled production agents.
5. Six months into deployment, your agent’s average cost-per-task has tripled. The most likely cause is:
a) The vendor raised prices
b) The team is sending more tokens per call than they realize — longer prompts, more tools, deeper retrieval
c) The model got worse
d) Volume increased
Correct Answer: b) Cost surprises in production almost always come from drift in what’s sent to the model, not from the model itself. Audit token-in / token-out per call before suspecting anything else.
Scoring
- 5 correct: Production-ready. You’re thinking about the right problems. Pick your first hard use case.
- 3–4 correct: Pilot-ready. The fundamentals are there; close the specific gaps before scaling.
- 1–2 correct: Foundation-building. Spend a quarter on observability, permissions, and one well-scoped pilot before greenlighting more.
- 0 correct: Hire someone, or hire a partner. The fastest path forward is bringing in someone who has done this once before.