Job: Import Movies

At a Glance

Import one or more raw movies for processing.

Description

The Import Movies job imports raw Cryo-EM movies into CryoSPARC for end-to-end processing. Importantly, the raw movie files that are imported by this job are not copied — they are merely linked (via a symbolic link) into the CryoSPARC project directory where this job is running. It is thus critical that raw movie files are not deleted or moved while they are being processed in CryoSPARC.

CryoSPARC is capable of processing raw data in the form of movies and micrographs. Data from modern direct electron detectors typically come in the form of a movie (multiple frames), while negative stain data are typically collected as a micrograph (one frame). In CryoSPARC, an exposure is the generic term referring to the entire data collection event for a single region of the grid and can be either a movie or a micrograph.

Inputs

This job does not accept any inputs.

Commonly Adjusted Parameters

If working with negative stain or phase plate data, please see the relevant pages for advice on importing your data to CryoSPARC:

Tutorial: Negative Stain Data

Tutorial: Phase Plate Data

Movies data path

The path in which movies are stored. Enter a path, or click on the folder icon to browse or paste the path specifying the location where the movies are stored. To select multiple files, enter a wildcard expression in the browse bar, e.g., /path/to/files/*.mrc, which will select all matching file types in the subfolder. Import Movies can accept movies in the .mrc, .mrc.bz2 , .tif or .eer formats.

Gain reference path

The path to the gain reference file, if available.

If you do not have a gain reference image available, first ask your microscope facility for help as it may be stored elsewhere in your data. Otherwise, there are tools which can estimate a gain reference after the fact from your raw data such as RELION’s relion_estimate_gain, linked in the References.

What is a gain reference?

Gain references are used to account for differences in the ability of each pixel to detect an electron. A given pixel may be more or less sensitive to electrons than its neighbors. This difference in sensitivity can lead to banding patterns or other artifacts that do not reflect the true sample image. Gain references are special images (typically recorded in a manufacturer-specific format and converted to .mrc) which are multiplied by the experimental image to correct for these artifacts.

Microscope parameters

Raw pixel size and Total exposure dose are typically selected during data collection, while Accelerating voltage and Spherical aberration are properties of the microscope. These parameters are essential and must be set accurately. If you do not know them, contact the facility at which your data were collected for help.

Negative stain data

If Negative Stain Data is on, this indicates that there are light particles on dark background (-1). If it's off, this indicates the movies have dark particles on light background (cryo-em data, +1). Negative stain data are rarely collected in movie format — if you have single-frame micrographs, Import Micrographs is the correct job to use.

Skip header check

Turned on by default in v4.2+, off by default in prior versions.

When Skip header check is turned off, each movie file's header is read by the job to ensure that all movies are of the same size, resolution, and frame count. The header check helps to detect corrupt files which otherwise may cause errors in downstream jobs, but also can take a long time due to the number of file system operations needed to read the headers.

When this parameter is turned off (i.e., when the header check is used), set the Number of CPUs to parallelize during header check parameter to parallelize reading of exposure headers.

Note that Reference Based Motion Correction and some other jobs require that movies have the same number of frames. Some users have run into problems when using these jobs because the header check was skipped — if you are not certain that all of your movies have the correct number of frames, turning the header check on will avoid these problems down the road.

EER parameters

EER Number of Fractions and EER Upsampling Factor are only used when importing movies in the Electron-event representation (EER) as designed by Guo and colleagues (2020). These parameters determine the number of fractions (roughly equivalent to frames in other formats) and the final resolution sampling of the movie. The defaults are suitable for most cases. For more information, see the EER section below.

Note that the default upsampling factor of 2 is equivalent to having collected a super-resolution movie. At this setting, we recommend performing Fourier cropping in subsequent motion correction jobs unless the data are expected to go past the physical Nyquist resolution of the camera.

Outputs

Imported movies

Imported movies are the main expected output and can be used in other CryoSPARC jobs.

Failed movies

Failed movies are movies that failed the header check and are likely an incorrect size or corrupt. Most often files end up in this output because the gain references or other files were accidentally included in the Movies data path wildcard. If the header check is skipped, this output will be empty. It is not necessary to repeat the Import Movies job if you do not need or want to include the movies that failed the header check; you can simply carry on processing with the Imported movies only.

Common Problems

The Import Movies job will also output thumbnails of the movies with the gain reference applied. It is generally a good idea to check these thumbnails to determine whether flipping or rotation has been applied as expected.

An example thumbnail from an Import Movies job. On the left, the gain reference has not been flipped, yielding thumbnails with large dark and light stripes. Turning on “Flip gain ref & defect file in Y?” produced the image on the right. Data from EMPIAR 10288 (Kumar et al. 2019).

Common Next Steps

Movies must be motion corrected before further processing, typically by CryoSPARC’s Patch Motion Correction job.

Electron-event representation (EER)

Modern direct electron detectors are capable of extraordinarily high frame rates (for example, the Falcon 4 detector has a hardware frame rate of 250 frames per second). However, recording an entire image frame at this frame rate would produce movies with impractically large file sizes. These movies would have frames with almost all pixels dark, almost entirely wasted space. Thus, in typical movie formats (like .tif or .mrc) a much slower frame rate is used and electron detection events are combined in each of these frames to produce fewer frames with more information per frame.

Instead of recording the entire frame as an image, the EER format records individual electron events by their position and time (Guo et al. 2020). In essence, this allows the movie to record at the full hardware frame rate while producing small movie files. Additionally, the position at which the electron struck the detector can be determined to sub-pixel accuracy, allowing for recording movies at greater than physical resolution, much like Super Resolution modes in other image formats.

Downstream processing still requires traditional images. Thus, the individual electron events are combined when EER files are decoded, into a number of fractions. These fractions contain all of the electron detection events for a given temporal segment of the movie, acting in much the same way as a frame. Higher settings for EER fractions result in potentially higher temporal resolution of sample motion, at the expense of substantially increased processing demands.

Given that EER format records electron events with sub-pixel accuracy, an upsampling factor can be used to decode the files into fractions that are more finely sampled than the physical detector. Guo and colleagues report that the detector is able to capture information at two to three times the physical Nyquist resolution. If a low upsampling factor is used (e.g. 1), the high resolution signal can be aliased to lower frequencies and degrade image quality. They therefore recommend using a high upsampling factor, even as high as 4. If any upsampling is used, we recommend the movies are then Fourier cropped back to physical pixel size (or, for samples expected to achieve Nyquist, a final super-resolution sampling of 2x Nyquist) during motion correction.

References

1. Guo, Hui, et al. "Electron-event representation data enable efficient cryoEM file storage with full preservation of spatial and temporal resolution." IUCrJ 7.5 (2020): 860-869.

2. https://relion.readthedocs.io/en/release-3.1/Reference/MovieCompression.html#gain-estimation

3. Kumar, K. et al. Structure of a Signaling Cannabinoid Receptor 1-G Protein Complex. Cell 176, 448-458.e12 (2019).