3D Refinement
Jump to the 3D Refinement Jobs
The ultimate goal of nearly all single-particle analysis projects is a high-quality 3D map of the target biomolecule. However, the raw data is a collection of noisy 2D images.
3D Refinement is the process by which CryoSPARC determines the 3D volume most likely to have produced the raw images. As such, it is arguably the most central process in cryo-EM image processing, and a deep understanding of 3D Refinement is invaluable for the cryo-EM practitioner. In this page, we aim to provide a welcoming introduction to the fundamental concepts of 3D Refinement and therefore cryo-EM reconstruction generally.
3D Refinement in cryo-EM is an example of the more general Expectation Maximization algorithm that is used across cryo-EM methods to reconstruct density. Reading the Expectation Maximization page before this one may be useful.
This topic has been an active field of research for decades; a complete summary would be impossible. The interested reader may wish to continue their study with Cryo-EM in molecular and cellular biology (Sabil 2022) for a short review, or Single-particle Cryo-EM of Biological Macromolecules (ed. Glaeser, Nogales, and Chiu; 2021) and works cited therein for a more thorough introduction.
Summary
The general process for refining and reconstructing a 3D object from 2D images is as follows:
Determine the projection direction of the 2D image.
Insert the image’s 2D Fourier transform into the appropriate central slice of the volume’s 3D Fourier transform
Repeat this process for each image. If the projection directions are more-or-less balanced, the volume’s 3D Fourier transform will be filled.
Recover the volume from the Fourier transform.
Aligning images to a volume
As the electron beam passes through the sample, it is absorbed and scattered. The resulting image is a projection of the 3D volume, corrupted by the contrast transfer function and noise.
In this page, we present projections as if they have high amplitude contrast. That is, as if the darkness of the image is directly related to the scattering potential of the 3D object. In reality, image formation in cryo-EM is more complicated. See the Contrast in Cryo-EM page for more information.
When presented with a single projection like the one below, it is generally not possible to determine the original 3D volume: there are infinitely many objects which could produce a single given projection.
However, with multiple projections of the same object it becomes easier to determine the most likely volume which gave rise to all of the projections.
With these twelve projections, it seems likely that the 3D object is a rubber duck. If we assume this, we can compare projections of the 3D volume to the particle image to find their pose — the rotation and translation applied to a given 2D image to match the reference.
Key point: In order to assign an image’s pose, we first must have an estimate of the 3D object to compare the images to. For more information on this process, see Expectation Maximization in Cryo-EM.
Finding an image’s pose
To find a particle’s pose, we compare a projection of the reference to the particle image. In an ideal world, if we have the pose exactly right the image will match the reference projection perfectly. We could, in theory, compare every rotation and translation of our reference to the particle image until we found the right pose.
In practice, this approach is not possible. First, it is not possible to check “every” pose — rotation and translation are both continuous variables with an infinite number of possible values. Second, generating a projection and comparing it to the particle image is computationally expensive, and comparing across all possible poses for every particle image in a dataset becomes prohibitively expensive. To make the problem computationally tractable, cryo-EM refinement methods generally take advantage of the fact that the error (i.e., degree of matching) between the reference projection and the particle image is expected to be smooth as a function of pose. Put another way, poses nearby to each other will have similar error.
This assumption allows for discretizing the pose space, so that for each particle image, the number of poses to check becomes limited. If we check a pose P and find it has some error E, we can assume that poses near P have an error close to E without checking them. Searching the complete pose space exhaustively, however, remains very computationally expensive.
CryoSPARC pose search: branch and bound
Suppose you were trying to find a coffee shop your friend told you about. You wouldn’t start by checking every street corner on earth until you found the right one. Instead, you might start by checking if the coffee shop is in a particular country, then a region, then a city, and so on until you found it. CryoSPARC uses a specialized algorithm called “branch and bound” to perform this same type of hierarchical search for particle poses (Punjani et al. 2017). This algorithm dramatically reduces the computational cost of finding an image's pose, and is one of the reasons that CryoSPARC refinements can be very fast.
Here we explain branch and bound using a simplified example in which we try to find the rotation which matches a 2D object and a 1D projection. The same principles apply for a full 3D refinement.
In the first branch and bound iteration, only a few poses are checked. This is typically referred to as a “coarse grid”, since in a real single-particle analysis pipeline the entire 5-D pose space would be sampled in a grid (evenly spaced in each dimension). At each candidate pose, the error between the reference in that pose and the target image is calculated.
Because we are using a coarse grid, we make a reduced number of comparisons between the reference and the target. However, calculating the true error between the reference and the target would be prohibitively expensive for even this reduced number. Here branch and bound makes another optimization. For each of these candidate poses, a special lower bound function that is guaranteed to always be less than the true error is calculated. This lower bound is designed to trade tightness for speed. In early iterations, a less tight (i.e., potentially significantly lower than the true error) but faster bound is used (see Punjani et al. 2017 Supplementary Materials for a mathematical description of the lower bound).
Next, we calculate the true error at the point with the best lower bound. This true error is too expensive to calculate for every candidate pose, but calculating it once per iteration is fast.
Since we know that no pose can have a lower error than the lower bound, any poses for which the lower bound is greater than the true error calculated at the best candidate pose can be eliminated.
Now we move to the next iteration. The poses are sampled more finely and a more accurate lower bound is used. Both of these changes incur additional cost and slow the iteration down, but this increased cost is offset by eliminating regions of the grid in the previous iteration.
This process repeats. At each iteration, the grid becomes finer and the lower bound more accurate, while more of the potential poses are eliminated from consideration.
Finally, the remaining region of pose space contains only one candidate — the particle’s pose has been found.
In CryoSPARC, this branch and bound algorithm is followed for every particle image, producing pose estimates for each.
Finding an image's pose corresponds to the Expectation step of Expectation Maximization, which is the general algorithm used for cryo-EM density refinement. Next, wemust create an updated volume from the particle images, the Maximization step. Unlike 2D Classification, where the 2D class averages are straightforward averages of the 2D images, it is not immediately obvious how we create a 3D volume from 2D images and poses. This process is called “3D reconstruction”, and is described in the next section.
Producing a volume from images
The Fourier Slice Theorem
If these concepts are unfamiliar to you, you may find a review of the Fourier transform helpful before proceeding. A good starting place might be this overview from Better Explained or this article from 3blue1brown.
The Fourier Slice Theorem tells us that
Projecting an image along some direction φ is equivalent to taking a slice through the center of the Fourier transform of that image, normal to φ.
For simplicity, consider again a 2D example and its Fourier transform:
If we project the duck along the Y axis, we reduce to a 1D signal. We can take the Fourier transform of that signal as well.
Note that we can represent the 1D Fourier transform either as a graph or as a strip of pixels, with darker pixels corresponding to higher values, just as we do with the 2D Fourier transform.
The key insight of the Fourier slice theorem comes when we compare the 1D Fourier transform of the projection of the image (the bottom-right graph) with the slice through the center of the 2D Fourier transform at Y = 0 (that is, the horizontal slice through the center of the 2D Fourier transform).
The central slice of the 2D Fourier transform is identical to the Fourier transform of the projection. We can thus complete our picture of the relationship between the 2D and 1D images and Fourier transforms:
An intuitive explanation
The Fourier Slice Theorem follows directly from the definitions of the Fourier transform and projection, and various derivations are available online. Instead of recapitulating the formal proof of the theorem, we present here a brief, intuitive explanation of why the central slice normal to the projection direction is equivalent to the Fourier transform of the projection.
Ignore, for the moment, the actual object which is projected. Consider a single wave and its Fourier transform (and recall that an image is the sum of many such waves):
Now suppose we project the image (the wave) along the Y-axis. The Fourier Slice Theorem tells us that the projection should be the same as the central slice through the Fourier transform. All values in this central slice are zero — this means the projection should be a flat line at zero as well.
Consider each column of pixels one by one. Although the sine wave in question here is moving diagonally, each column is itself sinusoidal:
Each position in the projection will therefore be the sum of a sine wave, and the sum of a sine wave is zero. It follows, then, that all waves which oscillate in any direction other than exactly perpendicular to the direction of projection will not contribute any signal to the final projected image. Such waves lie in the central slice of the Fourier transform, exactly as the theorem describes.
Backprojection
If taking a slice out of the center of a Fourier transform equates to projection, it stands to reason that putting a slice back in to the Fourier transform equates to doing the reverse, somehow adding information to a 3D volume using the 2D images. This operation is called “backprojection”, and is the means by which we can assemble information about a 3D object using its 2D projections.
Intuitively, backprojection can be understood as “smearing out” an image back into 3D space. When we do this with only one image, we end up with a prism-like object that has no more information than the single image did.
Note that the backprojected image is not a solid smear tracing the outline of the image. The middle (where the duck’s neck is) must have less total mass than the body to account for the difference in intensity in the image. With only a single image, there is no single right way to accomplish this — we don’t know anything about the volume aside from its projection in this one direction. In this case, the intensity in the neck region is reduced by adding a gap in the middle of the volume.
Each image gets smeared out according to its viewing direction:
Note that the backprojected volumes are always in the same reference frame, so they do not necessarily line up with their corresponding image, which is a projection along a unique direction.
Since each backprojected volume is smeared along the image’s distinct viewing direction, the smeared, useless direction is different for each image. These independent backprojections constrain the possible volume, since it is assumed that the same volume produced each image.
Thus, when we add together these four backprojected volumes, we recover a 3D Volume which is recognizably a duck. This process of adding together backprojections to recover the source 3D volume is called reconstruction. The more images we use, the more accurate our reconstruction becomes.
Per-particle scale
Another consideration when performing backprojection is that the absolute value of the pixels in an image are a function of both the electron potential of the 3D object (which we care about) and irrelevant facts about the environment around the particle, including ice thickness, nearby particles, microscope parameters, etc. However, in the final map, the value in a given voxel should relate directly to the electron potential in that region of space.
We account for these differences with the per-particle scale. The per-particle scale is a number relating the values in the local greyscale of the image to the greyscale of the volume. When each raw image is multiplied by its per-particle scale, the images are all on the same grayscale and can be backprojected to produce the volume.
3D Refinement Jobs
Job: Homogeneous RefinementJob: Heterogeneous RefinementJob: Non-Uniform RefinementJob: Homogeneous Reconstruction OnlyJob: Heterogeneous Reconstruction OnlyReferences
Single-Particle Cryo-EM of Biological Macromolecules. (IOP Publishing, 2021). doi:10.1088/978-0-7503-3039-8.
Kumar, K. et al. Structure of a Signaling Cannabinoid Receptor 1-G Protein Complex. Cell 176, 448-458.e12 (2019).
Penczek, P. A. Chapter One - Fundamentals of Three-Dimensional Reconstruction from Projections. in Methods in Enzymology (ed. Jensen, G. J.) vol. 482 1–33 (Academic Press, 2010).
Punjani, A., Rubinstein, J. L., Fleet, D. J. & Brubaker, M. A. cryoSPARC: algorithms for rapid unsupervised cryo-EM structure determination. Nat. Methods 14, 290 (2017).
Saibil, H. R. Cryo-EM in molecular and cellular biology. Molecular Cell 82, 274–284 (2022).
Last updated