LLM features need the same operational discipline as APIs: traces, budgets, and alerts when quality drifts.
Trace every call
Capture prompt version, model, tokens, latency, and tool outcomes. Without traces, debugging is guesswork.
Cost as a first-class signal
Set per-tenant and per-feature budgets. Surprise spend is usually a looping agent or an oversized context window.
Quality regressions
Compare weekly evaluation scores after prompt or model changes. Treat drops like failed deploys.