Summary: Matei Zaharia, a Berkeley computer science professor and Databricks co-founder known for creating Apache Spark, has been awarded the 2026 ACM Prize in Computing. The $250,000 prize recognizes his foundational work in distributed data systems and AI infrastructure. Zaharia is donating the prize to charity. He contends that AGI has already emerged, albeit in a form not widely appreciated, and believes AI shouldn’t be judged by human cognitive benchmarks.
Development of Apache Spark
Apache Spark, developed by Zaharia during his PhD at UC Berkeley in 2009, offered a quicker alternative to Hadoop MapReduce by moving intermediate computations to memory. This reduced processing time from hours to minutes, leading to Spark’s widespread adoption for analytical workloads. Zaharia’s work on Spark won the ACM Doctoral Dissertation Award in 2014 and laid the groundwork for Databricks, co-founded in 2013 by Zaharia and six colleagues. By December 2025, Databricks had a $134 billion valuation with a $5.4 billion revenue run rate. The ACM praised Zaharia’s visionary contributions to enabling large-scale machine learning and AI. Apache Spark, under Apache 2.0 license, has become a default framework for AI model releases, similar to Google’s recent Gemma 4 model family release.
Contributions with Delta Lake and MLflow
Zaharia’s innovation extended beyond Spark. As data infrastructure transitioned to the cloud, Zaharia co-developed Delta Lake, introducing ACID transactional semantics to cloud object stores, facilitating the data lakehouse architecture. This combines the advantages of data lakes and data warehouses. Databricks’ lakehouse architecture is a core commercial offering, adopted widely for enterprise data engineering. Additionally, Zaharia co-created MLflow to manage ML model operations, providing a structured framework for tracking experiments and deployments across various tools.
Recent Focus on AI Agents and DSPy
Zaharia’s current research focuses on AI systems reliability. He co-authored DSPy, an open-source framework that optimizes language model prompts and parameters, shifting from manual prompt engineering to improve AI production systems’ robustness. A related project, GEPA, aims to enhance multi-step AI workflows’ reliability. Zaharia’s career consistently applies systems thinking to data pipelines, deployment infrastructure, and now AI agent orchestration. This work has contributed significantly to the AI deployment ecosystem, now a major commercial market.
Perspective on AGI
The announcement’s highlight was Zaharia’s statement on AGI: “AGI is here already, it’s just not in a form that we appreciate.” He argues against comparing AI to human cognition, suggesting that AI’s capabilities differ from human intelligence. For instance, AI can rapidly consume and utilize extensive legal knowledge, unlike humans who require lengthy study. Zaharia posits that dismissing AI’s capabilities because they differ from human learning paths is arbitrary. This perspective challenges the conventional approach to defining and measuring AI progress.