Neural Radiance fields for Cryo-EM

Neural Radiance fields are all the rage in machine learning conferences right now. One could argue that they’re comparable to GANs when they first emerged.

With all the recent advances and hype over Neural Radiance fields, I figured I’d demonstrate one of the most exciting and impactful ways in which it could be used.

Cryo-EM background

Cryo-EM is a technique used to solve the 3D structure of protein macromolecules.

Proteins are small. Really small. In fact, proteins are so small that many are close in size (or smaller) than the wavelengths of visible light photons.

Because of this, an Electron Microscope that uses a beam of electrons can resolve finer details in microscopic samples than an optical light microscope.

The theoretical and experimental basis for Cryo-EM was established back in the 1970s. In a nutshell, the process involves freezing samples of a protein with liquid ethane, and then photographing them with an electron microscope. In 2017, Jacques Dubochet, Joachim Frank, and Richard Henderson won the Nobel prize in Chemistry for their pioneering work in Cryo-EM.

While this process of taking pictures was quite simple, the process of creating the 3D structure of the protein was more complicated.

Early computer vision approaches.

One of the early challenges in the field was detecting the particles in the Cryo-EM micrographs. Proteins are small, even by electron micrograph standards. Before one could try reconstructing the 3D structure of a protein, one needed to find the particles in the micrograph.

This object detection problem was an early application of Haar Cascades (Viola & Jones), as described in Detecting particles in cryo-EM micrographs using learned features (2004). As is the case now, a lot of value can be produced by taking a state-of-the-art machine learning technique and simply applying it in a new domain.

Haar Cascades mae selecting the particles much easier, but the much more daunting task remained: reconstructing the 3D structure. In the coming decades, countless then-SOTA algorithms would test their mettle in this domain (e.g., 3D reconstruction of protein macromolecules using Cryo-EM images). The holy grail of protein reconstruction was atomic resolution (4 angstroms) of protein structure.

Bringing NeRFs into the space

UPDATE 10/12/2021: While this Tutorial goes into the basics, I want to bring everyone’s attention to the recently released CryoDRGN2 attacks the challenging problem of reconstructing protein structure and pose from a “multiview” set of cryo-EM density images. It is unique among NeRF-style papers as it works in the Fourier domain.

Cited as:

@article{mcateer2021cryonerf,
    title = "Neural Radiance fields for Cryo-EM",
    author = "McAteer, Matthew",
    journal = "matthewmcateer.me",
    year = "2021",
    url = "https://matthewmcateer.me/blog/cryo-em-nerfs/"
}

If you notice mistakes and errors in this post, don’t hesitate to contact me at [contact at matthewmcateer dot me] and I will be very happy to correct them right away! Alternatily, you can follow me on Twitter and reach out to me there.

See you in the next post 😄