Google DeepMind’s efforts with AlphaFold have been remarkable, yet they come with significant computational costs. Considering this, Apple researchers embarked on creating an alternative approach to utilize AI for predicting protein 3D structures, and the results are encouraging. Here are the specifics.
If you’re unfamiliar with AlphaFold, it’s Google DeepMind’s revolutionary AI model that predicts the 3D shape of proteins based on their amino acid sequences. This has proven particularly beneficial for the development of more effective medications, along with entirely new materials.
Until recently, this was a highly challenging issue. Estimating the three-dimensional atomic formation of a single protein could span months or even years.
However, with the advent of AlphaFold, now AlphaFold2, as well as other advanced models such as RoseTTAFold and ESMFold, this prediction time can be reduced to mere hours or even minutes, contingent on the hardware available.
Each model employs distinct techniques and frameworks to achieve such impressive accuracy; however, they generally entail extremely expensive computations, and their frameworks follow a rigid structure.
According to Apple’s researchers:
“Long-established models for protein folding such as AlphaFold2 and RoseTTAFold have garnered exceptional accuracy through meticulously designed architectures that incorporate computation-heavy, domain-specific functionalities for tasks like multiple sequence alignments (MSAs) of amino acid sequences, pair representations, and triangle updates. These design selections (MSA, pair representations, triangular updates, etc.) aim to embed our current comprehension of the fundamental structure generation processes into these models, rather than allowing the models to learn this directly from data, which could be advantageous for numerous reasons.”
### Introducing Apple’s SimpleFold
In their model proposal, instead of depending on “MSA, pairwise interaction maps, triangular updates, or any other equivariant geometric modules,” Apple utilizes what are known as flow matching models, introduced in 2023 that have gained significant traction for text-to-image and text-to-3D applications.
Essentially, flow matching models represent an advancement from the diffusion models discussed in a prior article. Rather than merely iteratively minimizing noise from an initial image, they learn a smoother trajectory that transforms random noise directly into a completed image in one step.
Moreover, as this method bypasses many denoising processes, it proves to be less computationally intensive, yielding results more quickly.
Apple researchers trained SimpleFold at various sizes, specifically 100M, 360M, 700M, 1.1B, 1.6B, and 3B parameters, and evaluated them based on “two widely recognized protein structure prediction benchmarks: CAMEO22 and CASP14, which are stringent tests for generalization, robustness, and atomic-level precision in folding models.”
The outcomes were quite promising:
“Despite its straightforwardness, SimpleFold delivers competitive performance relative to these baselines. In both benchmarks, SimpleFold consistently surpasses ESMFlow, another flow-matching model designed with ESM embeddings. On CAMEO22, SimpleFold yields results comparable to the top folding models (e.g., ESMFold, RoseTTAFold2, and AlphaFold2). Notably, SimpleFold attains over 95% of the performance of RoseTTAFold2/AlphaFold2 across most metrics without utilizing costly and heuristic triangle attention and MSA.”
And
“To ensure thoroughness, we present the results of SimpleFold across various model sizes. The smallest model, SimpleFold-100M, demonstrates competitive performance while maintaining efficiency in both training and inference. Specifically, SimpleFold obtains over 90% of the ESMFold performance on CAMEO22, illustrating the effectiveness of constructing a folding model utilizing general-purpose architectural components.”
They also observed performance enhancements correlated with scaling, indicating that larger models with extensive training data consistently yield superior folding performance, particularly on the most demanding benchmarks.
Ultimately, they indicate that SimpleFold is merely an initial effort, expressing their hope that it “serves as a catalyst for the community to develop efficient and powerful protein generative models.”
You can access the complete study on arXiv.
Read More
