LLM-powered systems are increasingly being integrated into production, but their non-deterministic nature poses challenges not typically faced in traditional software development. Testing changes, diagnosing failures, and confidently releasing updates are difficult, necessitating new evaluation tools tailored to LLMs’ characteristics.
Comet is a platform focused on agent-based systems, optimizing prompts, tools, and workflows as components for evaluation and enhancement over time.
Gideon Mendels, co-founder and CEO of Comet, who has experience at Google with hate speech and deception detection as well as founding GroupWise for processing chats, discusses with Kevin Ball the intersection of agent development with software engineering and ML, the importance of eVals for AI teams, prompt optimization as a search challenge, and ongoing improvements for production agents.
Kevin Ball, known as KBall, is Mento’s VP of Engineering and an independent coach for engineers. He co-founded two companies, initiated the San Diego JavaScript meetup, and organizes the AI inaction group through Latent Space.
Full Disclosure: This episode is sponsored by Comet.
Please click here to see the transcript of this episode.