CryoSPARC Guide
  • About CryoSPARC
  • Current Version
  • Licensing
    • Non-commercial license agreement
  • Setup, Configuration and Management
    • CryoSPARC Architecture and System Requirements
    • CryoSPARC Installation Prerequisites
    • How to Download, Install and Configure
      • Obtaining A License ID
      • Downloading and Installing CryoSPARC
      • CryoSPARC Cluster Integration Script Examples
      • Accessing the CryoSPARC User Interface
    • Deploying CryoSPARC on AWS
      • Performance Benchmarks
    • Using CryoSPARC with Cluster Management Software
    • Software Updates and Patches
    • Management and Monitoring
      • Environment variables
      • (Optional) Hosting CryoSPARC Through a Reverse Proxy
      • cryosparcm reference
      • cryosparcm cli reference
      • cryosparcw reference
    • Software System Guides
      • Guide: Updating to CryoSPARC v4
      • Guide: Installation Testing with cryosparcm test
      • Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)
      • Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)
      • Guide: Performance Benchmarking (v4.3+)
      • Guide: Download Error Reports
      • Guide: Maintenance Mode and Configurable User Facing Messages
      • Guide: User Management
      • Guide: Multi-user Unix Permissions and Data Access Control
      • Guide: Lane Assignments and Restrictions
      • Guide: Queuing Directly to a GPU
      • Guide: Priority Job Queuing
      • Guide: Configuring Custom Variables for Cluster Job Submission Scripts
      • Guide: SSD Particle Caching in CryoSPARC
      • Guide: Data Management in CryoSPARC (v4.0+)
      • Guide: Data Cleanup (v4.3+)
      • Guide: Reduce Database Size (v4.3+)
      • Guide: Data Management in CryoSPARC (≤v3.3)
      • Guide: CryoSPARC Live Session Data Management
      • Guide: Manipulating .cs Files Created By CryoSPARC
      • Guide: Migrating your CryoSPARC Instance
      • Guide: EMDB-friendly XML file for FSC plots
    • Troubleshooting
  • Application Guide (v4.0+)
    • A Tour of the CryoSPARC Interface
    • Browsing the CryoSPARC Instance
    • Projects, Workspaces and Live Sessions
    • Jobs
    • Job Views: Cards, Tree, and Table
    • Creating and Running Jobs
    • Low Level Results Interface
    • Filters and Sorting
    • View Options
    • Tags
    • Flat vs Hierarchical Navigation
    • File Browser
    • Blueprints
    • Workflows
    • Inspecting Data
    • Managing Jobs
    • Interactive Jobs
    • Upload Local Files
    • Managing Data
    • Downloading and Exporting Data
    • Instance Management
    • Admin Panel
  • Cryo-EM Foundations
    • Image Formation
      • Contrast in Cryo-EM
      • Waves as Vectors
      • Aliasing
  • Expectation Maximization in Cryo-EM
  • Processing Data in cryoSPARC
    • Get Started with CryoSPARC: Introductory Tutorial (v4.0+)
    • Tutorial Videos
    • All Job Types in CryoSPARC
      • Import
        • Job: Import Movies
        • Job: Import Micrographs
        • Job: Import Particle Stack
        • Job: Import 3D Volumes
        • Job: Import Templates
        • Job: Import Result Group
        • Job: Import Beam Shift
      • Motion Correction
        • Job: Patch Motion Correction
        • Job: Full-Frame Motion Correction
        • Job: Local Motion Correction
        • Job: MotionCor2 (Wrapper) (BETA)
        • Job: Reference Based Motion Correction (BETA)
      • CTF Estimation
        • Job: Patch CTF Estimation
        • Job: Patch CTF Extraction
        • Job: CTFFIND4 (Wrapper)
        • Job: Gctf (Wrapper) (Legacy)
      • Exposure Curation
        • Job: Micrograph Denoiser (BETA)
        • Job: Micrograph Junk Detector (BETA)
        • Interactive Job: Manually Curate Exposures
      • Particle Picking
        • Interactive Job: Manual Picker
        • Job: Blob Picker
        • Job: Template Picker
        • Job: Filament Tracer
        • Job: Blob Picker Tuner
        • Interactive Job: Inspect Particle Picks
        • Job: Create Templates
      • Extraction
        • Job: Extract from Micrographs
        • Job: Downsample Particles
        • Job: Restack Particles
      • Deep Picking
        • Guideline for Supervised Particle Picking using Deep Learning Models
        • Deep Network Particle Picker
          • T20S Proteasome: Deep Particle Picking Tutorial
          • Job: Deep Picker Train and Job: Deep Picker Inference
        • Topaz (Bepler, et al)
          • T20S Proteasome: Topaz Particle Picking Tutorial
          • T20S Proteasome: Topaz Micrograph Denoising Tutorial
          • Job: Topaz Train and Job: Topaz Cross Validation
          • Job: Topaz Extract
          • Job: Topaz Denoise
      • Particle Curation
        • Job: 2D Classification
        • Interactive Job: Select 2D Classes
        • Job: Reference Based Auto Select 2D (BETA)
        • Job: Reconstruct 2D Classes
        • Job: Rebalance 2D Classes
        • Job: Class Probability Filter (Legacy)
        • Job: Rebalance Orientations
        • Job: Subset Particles by Statistic
      • 3D Reconstruction
        • Job: Ab-Initio Reconstruction
      • 3D Refinement
        • Job: Homogeneous Refinement
        • Job: Heterogeneous Refinement
        • Job: Non-Uniform Refinement
        • Job: Homogeneous Reconstruction Only
        • Job: Heterogeneous Reconstruction Only
        • Job: Homogeneous Refinement (Legacy)
        • Job: Non-uniform Refinement (Legacy)
      • CTF Refinement
        • Job: Global CTF Refinement
        • Job: Local CTF Refinement
        • Job: Exposure Group Utilities
      • Conformational Variability
        • Job: 3D Variability
        • Job: 3D Variability Display
        • Job: 3D Classification
        • Job: Regroup 3D Classes
        • Job: Reference Based Auto Select 3D (BETA)
        • Job: 3D Flexible Refinement (3DFlex) (BETA)
      • Postprocessing
        • Job: Sharpening Tools
        • Job: DeepEMhancer (Wrapper)
        • Job: Validation (FSC)
        • Job: Local Resolution Estimation
        • Job: Local Filtering
        • Job: ResLog Analysis
        • Job: ThreeDFSC (Wrapper) (Legacy)
      • Local Refinement
        • Job: Local Refinement
        • Job: Particle Subtraction
        • Job: Local Refinement (Legacy)
      • Helical Reconstruction
        • Helical symmetry in CryoSPARC
        • Job: Helical Refinement
        • Job: Symmetry search utility
        • Job: Average Power Spectra
      • Utilities
        • Job: Exposure Sets Tool
        • Job: Exposure Tools
        • Job: Generate Micrograph Thumbnails
        • Job: Cache Particles on SSD
        • Job: Check for Corrupt Particles
        • Job: Particle Sets Tool
        • Job: Reassign Particles to Micrographs
        • Job: Remove Duplicate Particles
        • Job: Symmetry Expansion
        • Job: Volume Tools
        • Job: Volume Alignment Tools
        • Job: Align 3D maps
        • Job: Split Volumes Group
        • Job: Orientation Diagnostics
      • Simulations
        • Job: Simulate Data (GPU)
        • Job: Simulate Data (Legacy)
    • CryoSPARC Tools
    • Data Processing Tutorials
      • Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)
      • Case Study: DkTx-bound TRPV1 (EMPIAR-10059)
      • Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)
      • Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)
      • Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)
      • Case Study: Exploratory data processing by Oliver Clarke
      • Tutorial: Tips for Membrane Protein Structures
      • Tutorial: Common CryoSPARC Plots
      • Tutorial: Negative Stain Data
      • Tutorial: Phase Plate Data
      • Tutorial: EER File Support
      • Tutorial: EPU AFIS Beam Shift Import
      • Tutorial: Patch Motion and Patch CTF
      • Tutorial: Float16 Support
      • Tutorial: Particle Picking Calibration
      • Tutorial: Blob Picker Tuner
      • Tutorial: Helical Processing using EMPIAR-10031 (MAVS)
      • Tutorial: Maximum Box Sizes for Refinement
      • Tutorial: CTF Refinement
      • Tutorial: Ewald Sphere Correction
      • Tutorial: Symmetry Relaxation
      • Tutorial: Orientation Diagnostics
      • Tutorial: BILD files in CryoSPARC v4.4+
      • Tutorial: Mask Creation
      • Case Study: Yeast U4/U6.U5 tri-snRNP
      • Tutorial: 3D Classification
      • Tutorial: 3D Variability Analysis (Part One)
      • Tutorial: 3D Variability Analysis (Part Two)
      • Tutorial: 3D Flexible Refinement
        • Installing 3DFlex Dependencies (v4.1–v4.3)
      • Tutorial: 3D Flex Mesh Preparation
    • Webinar Recordings
  • Real-time processing in cryoSPARC Live
    • About CryoSPARC Live
    • Prerequisites and Compute Resources Setup
    • How to Access cryoSPARC Live
    • UI Overview
    • New Live Session: Start to Finish Guide
    • CryoSPARC Live Tutorial Videos
    • Live Jobs and Session-Level Functions
    • Performance Metrics
    • Managing a CryoSPARC Live Session from the CLI
    • FAQs and Troubleshooting
  • Guides for v3
    • v3 User Interface Guide
      • Dashboard
      • Project and Workspace Management
      • Create and Build Jobs
      • Queue Job, Inspect Job and Other Job Actions
      • View and Download Results
      • Job Relationships
      • Resource Manager
      • User Management
    • Tutorial: Job Builder
    • Get Started with CryoSPARC: Introductory Tutorial (v3)
    • Tutorial: Manually Curate Exposures (v3)
  • Resources
    • Questions and Support
Powered by GitBook
On this page
  • At a Glance
  • Description
  • Inputs
  • Commonly Adjusted Parameters
  • Movies data path
  • Gain reference path
  • Microscope parameters
  • Negative stain data
  • Skip header check
  • EER parameters
  • Outputs
  • Imported movies
  • Failed movies
  • Common Problems
  • Common Next Steps
  • Electron-event representation (EER)
  • References
  1. Processing Data in cryoSPARC
  2. All Job Types in CryoSPARC
  3. Import

Job: Import Movies

At a Glance

Import one or more raw movies for processing.

Description

The Import Movies job imports raw Cryo-EM movies into CryoSPARC for end-to-end processing. Importantly, the raw movie files that are imported by this job are not copied — they are merely linked (via a symbolic link) into the CryoSPARC project directory where this job is running. It is thus critical that raw movie files are not deleted or moved while they are being processed in CryoSPARC.

CryoSPARC is capable of processing raw data in the form of movies and micrographs. Data from modern direct electron detectors typically come in the form of a movie (multiple frames), while negative stain data are typically collected as a micrograph (one frame). In CryoSPARC, an exposure is the generic term referring to the entire data collection event for a single region of the grid and can be either a movie or a micrograph.

Inputs

This job does not accept any inputs.

Commonly Adjusted Parameters

If working with negative stain or phase plate data, please see the relevant pages for advice on importing your data to CryoSPARC:

Tutorial: Negative Stain Data

Tutorial: Phase Plate Data

Movies data path

The path in which movies are stored. Enter a path, or click on the folder icon to browse or paste the path specifying the location where the movies are stored. To select multiple files, enter a wildcard expression in the browse bar, e.g., /path/to/files/*.mrc, which will select all matching file types in the subfolder. Import Movies can accept movies in the .mrc, .mrc.bz2 , .tif or .eer formats.

Gain reference path

The path to the gain reference file, if available.

What is a gain reference?

Gain references are used to account for differences in the ability of each pixel to detect an electron. A given pixel may be more or less sensitive to electrons than its neighbors. This difference in sensitivity can lead to banding patterns or other artifacts that do not reflect the true sample image. Gain references are special images (typically recorded in a manufacturer-specific format and converted to .mrc) which are multiplied by the experimental image to correct for these artifacts.

Microscope parameters

Raw pixel size and Total exposure dose are typically selected during data collection, while Accelerating voltage and Spherical aberration are properties of the microscope. These parameters are essential and must be set accurately. If you do not know them, contact the facility at which your data were collected for help.

Negative stain data

If Negative Stain Data is on, this indicates that there are light particles on dark background (-1). If it's off, this indicates the movies have dark particles on light background (cryo-em data, +1). Negative stain data are rarely collected in movie format — if you have single-frame micrographs, Import Micrographs is the correct job to use.

Skip header check

Turned on by default in v4.2+, off by default in prior versions.

When Skip header check is turned off, each movie file's header is read by the job to ensure that all movies are of the same size, resolution, and frame count. The header check helps to detect corrupt files which otherwise may cause errors in downstream jobs, but also can take a long time due to the number of file system operations needed to read the headers.

When this parameter is turned off (i.e., when the header check is used), set the Number of CPUs to parallelize during header check parameter to parallelize reading of exposure headers.

EER parameters

Note that the default upsampling factor of 2 is equivalent to having collected a super-resolution movie. At this setting, we recommend performing Fourier cropping in subsequent motion correction jobs unless the data are expected to go past the physical Nyquist resolution of the camera.

Outputs

Imported movies

Imported movies are the main expected output and can be used in other CryoSPARC jobs.

Failed movies

Failed movies are movies that failed the header check and are likely an incorrect size or corrupt. Most often files end up in this output because the gain references or other files were accidentally included in the Movies data path wildcard. If the header check is skipped, this output will be empty. It is not necessary to repeat the Import Movies job if you do not need or want to include the movies that failed the header check; you can simply carry on processing with the Imported movies only.

Common Problems

The Import Movies job will also output thumbnails of the movies with the gain reference applied. It is generally a good idea to check these thumbnails to determine whether flipping or rotation has been applied as expected.

Common Next Steps

Electron-event representation (EER)

Modern direct electron detectors are capable of extraordinarily high frame rates (for example, the Falcon 4 detector has a hardware frame rate of 250 frames per second). However, recording an entire image frame at this frame rate would produce movies with impractically large file sizes. These movies would have frames with almost all pixels dark, almost entirely wasted space. Thus, in typical movie formats (like .tif or .mrc) a much slower frame rate is used and electron detection events are combined in each of these frames to produce fewer frames with more information per frame.

Instead of recording the entire frame as an image, the EER format records individual electron events by their position and time (Guo et al. 2020). In essence, this allows the movie to record at the full hardware frame rate while producing small movie files. Additionally, the position at which the electron struck the detector can be determined to sub-pixel accuracy, allowing for recording movies at greater than physical resolution, much like Super Resolution modes in other image formats.

Downstream processing still requires traditional images. Thus, the individual electron events are combined when EER files are decoded, into a number of fractions. These fractions contain all of the electron detection events for a given temporal segment of the movie, acting in much the same way as a frame. Higher settings for EER fractions result in potentially higher temporal resolution of sample motion, at the expense of substantially increased processing demands.

Given that EER format records electron events with sub-pixel accuracy, an upsampling factor can be used to decode the files into fractions that are more finely sampled than the physical detector. Guo and colleagues report that the detector is able to capture information at two to three times the physical Nyquist resolution. If a low upsampling factor is used (e.g. 1), the high resolution signal can be aliased to lower frequencies and degrade image quality. They therefore recommend using a high upsampling factor, even as high as 4. If any upsampling is used, we recommend the movies are then Fourier cropped back to physical pixel size (or, for samples expected to achieve Nyquist, a final super-resolution sampling of 2x Nyquist) during motion correction.

References

  1. Guo, Hui, et al. "Electron-event representation data enable efficient cryoEM file storage with full preservation of spatial and temporal resolution." IUCrJ 7.5 (2020): 860-869.

  2. Kumar, K. et al. Structure of a Signaling Cannabinoid Receptor 1-G Protein Complex. Cell 176, 448-458.e12 (2019).

PreviousImportNextJob: Import Micrographs

Last updated 1 year ago

If you do not have a gain reference image available, first ask your microscope facility for help as it may be stored elsewhere in your data. Otherwise, there are tools which can estimate a gain reference after the fact from your raw data such as RELION’s relion_estimate_gain, linked in the .

Note that Reference Based Motion Correction and some other jobs require that movies have the same number of frames. when using these jobs because the header check was skipped — if you are not certain that all of your movies have the correct number of frames, turning the header check on will avoid these problems down the road.

EER Number of Fractions and EER Upsampling Factor are only used when importing movies in the as designed by Guo and colleagues (2020). These parameters determine the number of fractions (roughly equivalent to frames in other formats) and the final resolution sampling of the movie. The defaults are suitable for most cases. For more information, see the EER section below.

Movies must be motion corrected before further processing, typically by CryoSPARC’s job.

Some users have run into problems
Patch Motion Correction
https://relion.readthedocs.io/en/release-3.1/Reference/MovieCompression.html#gain-estimation
References
Electron-event representation (EER)
An example thumbnail from an Import Movies job. On the left, the gain reference has not been flipped, yielding thumbnails with large dark and light stripes. Turning on “Flip gain ref & defect file in Y?” produced the image on the right. Data from EMPIAR 10288 (Kumar et al. 2019).