LLM-powered systems are steadily moving into production, presenting teams with challenges not commonly encountered by traditional software practices. Models and agents are non-deterministic, making it difficult to test changes, reason about failures, and confidently ship updates. This has created a need for new evaluation tools specifically designed around the properties of LLMs.
Comet is a platform integrating Roots and MLOps into the rapidly evolving world of agent-based systems by treating prompts, tools, and workflows as optimizable components that can be evaluated and improved over time.
Gideon Mendels is the co-founder and CEO of Comet. He previously worked at Google on hate speech and deception detection and founded GroupWise, which trained and deployed NLP models processing billions of chats. In this episode, Gideon joins Kevin Ball to discuss how agent development sits between software engineering and ML, why eVals are the missing foundation for most AI teams, prompt optimization as a search problem, and the future for continuously improving agents in production.
Full Disclosure: This episode is sponsored by Comet.
Kevin Ball, also known as KBall, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI inaction discussion group through Latent Space.
Please click here to see the transcript of this episode.