核心观点
@ds3638 (Dhruv Singh): Evals are dead. Or more precisely:
traditional eval-driven development doesn’t scale.
Static evals were useful when agents were short-lived and bounded, but once agents are running for hours and taking thousands of actions + operating autonomously, evals alone stop being enough.
At that point pass/fail is too coarse. Simulation misses too much of what happens in prod and model capabilities are moving faster than eval infra can keep up.
What we run instead: observability-driven development.
- deploy with tight guardrails
- collect prod trajectories
- cluster behavior to discover patterns + failure modes
- specialize workers for narrower tasks
- tune thresholds until behavior is reliably within bounds
Can you see what your agents are doing? Can you detect drifts before they cause damage?
This is an important shift in how we build AI systems. Evals still matter but observability is becoming the foundation for prod-ready agents.
Thanks Sunny Bakhda (@honeyhiveai founding engineer) for a great talk at @aicouncilconf 🖼️ https://pbs.twimg.com/media/HIS1kCXX0AAPQFD.jpg 📅 Thu May 14 16:47:58 +0000 2026 🔗 https://x.com/ds3638/status/2054966962803839347 ❤️ 49 🔁 6 💬 4
解读
用 Claude 生成完整商业计划，替代高价咨询
原文由 @Whizz_ai 发布于 X。解读由 SOTA Sync 生成。