Assessing software quality, performance, and effectiveness through evaluations is crucial during development. Techniques like code reviews and automated testing are common for identifying bugs, ensuring requirement compliance, and measuring software reliability.
Yet, evaluating large language models (LLMs) is uniquely challenging due to their complexity, versatility, and potential for unpredictable behavior.
Ankur Goyal, CEO and Founder of Braintrust Data, offers an end-to-end platform for AI application development, emphasizing robust and iterative LLM development. Formerly, Ankur founded Impira, which was acquired by Figma, where he led the AI team. He discusses Braintrust and the challenges of developing evaluations in non-deterministic contexts.
Sean, an academic, startup founder, and Googler, has authored works spanning AI to quantum computing. As AI Entrepreneur in Residence at Confluent, he focuses on AI strategy and thought leadership. Connect with Sean on LinkedIn.
[Access the episode transcript here.]