CryoSPARC Guide
  • About CryoSPARC
  • Current Version
  • Licensing
    • Non-commercial license agreement
  • Setup, Configuration and Management
    • CryoSPARC Architecture and System Requirements
    • CryoSPARC Installation Prerequisites
    • How to Download, Install and Configure
      • Obtaining A License ID
      • Downloading and Installing CryoSPARC
      • CryoSPARC Cluster Integration Script Examples
      • Accessing the CryoSPARC User Interface
    • Deploying CryoSPARC on AWS
      • Performance Benchmarks
    • Using CryoSPARC with Cluster Management Software
    • Software Updates and Patches
    • Management and Monitoring
      • Environment variables
      • (Optional) Hosting CryoSPARC Through a Reverse Proxy
      • cryosparcm reference
      • cryosparcm cli reference
      • cryosparcw reference
    • Software System Guides
      • Guide: Updating to CryoSPARC v4
      • Guide: Installation Testing with cryosparcm test
      • Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)
      • Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)
      • Guide: Performance Benchmarking (v4.3+)
      • Guide: Download Error Reports
      • Guide: Maintenance Mode and Configurable User Facing Messages
      • Guide: User Management
      • Guide: Multi-user Unix Permissions and Data Access Control
      • Guide: Lane Assignments and Restrictions
      • Guide: Queuing Directly to a GPU
      • Guide: Priority Job Queuing
      • Guide: Configuring Custom Variables for Cluster Job Submission Scripts
      • Guide: SSD Particle Caching in CryoSPARC
      • Guide: Data Management in CryoSPARC (v4.0+)
      • Guide: Data Cleanup (v4.3+)
      • Guide: Reduce Database Size (v4.3+)
      • Guide: Data Management in CryoSPARC (≤v3.3)
      • Guide: CryoSPARC Live Session Data Management
      • Guide: Manipulating .cs Files Created By CryoSPARC
      • Guide: Migrating your CryoSPARC Instance
      • Guide: EMDB-friendly XML file for FSC plots
    • Troubleshooting
  • Application Guide (v4.0+)
    • A Tour of the CryoSPARC Interface
    • Browsing the CryoSPARC Instance
    • Projects, Workspaces and Live Sessions
    • Jobs
    • Job Views: Cards, Tree, and Table
    • Creating and Running Jobs
    • Low Level Results Interface
    • Filters and Sorting
    • View Options
    • Tags
    • Flat vs Hierarchical Navigation
    • File Browser
    • Blueprints
    • Workflows
    • Inspecting Data
    • Managing Jobs
    • Interactive Jobs
    • Upload Local Files
    • Managing Data
    • Downloading and Exporting Data
    • Instance Management
    • Admin Panel
  • Cryo-EM Foundations
    • Image Formation
      • Contrast in Cryo-EM
      • Waves as Vectors
      • Aliasing
  • Expectation Maximization in Cryo-EM
  • Processing Data in cryoSPARC
    • Get Started with CryoSPARC: Introductory Tutorial (v4.0+)
    • Tutorial Videos
    • All Job Types in CryoSPARC
      • Import
        • Job: Import Movies
        • Job: Import Micrographs
        • Job: Import Particle Stack
        • Job: Import 3D Volumes
        • Job: Import Templates
        • Job: Import Result Group
        • Job: Import Beam Shift
      • Motion Correction
        • Job: Patch Motion Correction
        • Job: Full-Frame Motion Correction
        • Job: Local Motion Correction
        • Job: MotionCor2 (Wrapper) (BETA)
        • Job: Reference Based Motion Correction (BETA)
      • CTF Estimation
        • Job: Patch CTF Estimation
        • Job: Patch CTF Extraction
        • Job: CTFFIND4 (Wrapper)
        • Job: Gctf (Wrapper) (Legacy)
      • Exposure Curation
        • Job: Micrograph Denoiser (BETA)
        • Job: Micrograph Junk Detector (BETA)
        • Interactive Job: Manually Curate Exposures
      • Particle Picking
        • Interactive Job: Manual Picker
        • Job: Blob Picker
        • Job: Template Picker
        • Job: Filament Tracer
        • Job: Blob Picker Tuner
        • Interactive Job: Inspect Particle Picks
        • Job: Create Templates
      • Extraction
        • Job: Extract from Micrographs
        • Job: Downsample Particles
        • Job: Restack Particles
      • Deep Picking
        • Guideline for Supervised Particle Picking using Deep Learning Models
        • Deep Network Particle Picker
          • T20S Proteasome: Deep Particle Picking Tutorial
          • Job: Deep Picker Train and Job: Deep Picker Inference
        • Topaz (Bepler, et al)
          • T20S Proteasome: Topaz Particle Picking Tutorial
          • T20S Proteasome: Topaz Micrograph Denoising Tutorial
          • Job: Topaz Train and Job: Topaz Cross Validation
          • Job: Topaz Extract
          • Job: Topaz Denoise
      • Particle Curation
        • Job: 2D Classification
        • Interactive Job: Select 2D Classes
        • Job: Reference Based Auto Select 2D (BETA)
        • Job: Reconstruct 2D Classes
        • Job: Rebalance 2D Classes
        • Job: Class Probability Filter (Legacy)
        • Job: Rebalance Orientations
        • Job: Subset Particles by Statistic
      • 3D Reconstruction
        • Job: Ab-Initio Reconstruction
      • 3D Refinement
        • Job: Homogeneous Refinement
        • Job: Heterogeneous Refinement
        • Job: Non-Uniform Refinement
        • Job: Homogeneous Reconstruction Only
        • Job: Heterogeneous Reconstruction Only
        • Job: Homogeneous Refinement (Legacy)
        • Job: Non-uniform Refinement (Legacy)
      • CTF Refinement
        • Job: Global CTF Refinement
        • Job: Local CTF Refinement
        • Job: Exposure Group Utilities
      • Conformational Variability
        • Job: 3D Variability
        • Job: 3D Variability Display
        • Job: 3D Classification
        • Job: Regroup 3D Classes
        • Job: Reference Based Auto Select 3D (BETA)
        • Job: 3D Flexible Refinement (3DFlex) (BETA)
      • Postprocessing
        • Job: Sharpening Tools
        • Job: DeepEMhancer (Wrapper)
        • Job: Validation (FSC)
        • Job: Local Resolution Estimation
        • Job: Local Filtering
        • Job: ResLog Analysis
        • Job: ThreeDFSC (Wrapper) (Legacy)
      • Local Refinement
        • Job: Local Refinement
        • Job: Particle Subtraction
        • Job: Local Refinement (Legacy)
      • Helical Reconstruction
        • Helical symmetry in CryoSPARC
        • Job: Helical Refinement
        • Job: Symmetry search utility
        • Job: Average Power Spectra
      • Utilities
        • Job: Exposure Sets Tool
        • Job: Exposure Tools
        • Job: Generate Micrograph Thumbnails
        • Job: Cache Particles on SSD
        • Job: Check for Corrupt Particles
        • Job: Particle Sets Tool
        • Job: Reassign Particles to Micrographs
        • Job: Remove Duplicate Particles
        • Job: Symmetry Expansion
        • Job: Volume Tools
        • Job: Volume Alignment Tools
        • Job: Align 3D maps
        • Job: Split Volumes Group
        • Job: Orientation Diagnostics
      • Simulations
        • Job: Simulate Data (GPU)
        • Job: Simulate Data (Legacy)
    • CryoSPARC Tools
    • Data Processing Tutorials
      • Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)
      • Case Study: DkTx-bound TRPV1 (EMPIAR-10059)
      • Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)
      • Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)
      • Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)
      • Case Study: Exploratory data processing by Oliver Clarke
      • Tutorial: Tips for Membrane Protein Structures
      • Tutorial: Common CryoSPARC Plots
      • Tutorial: Negative Stain Data
      • Tutorial: Phase Plate Data
      • Tutorial: EER File Support
      • Tutorial: EPU AFIS Beam Shift Import
      • Tutorial: Patch Motion and Patch CTF
      • Tutorial: Float16 Support
      • Tutorial: Particle Picking Calibration
      • Tutorial: Blob Picker Tuner
      • Tutorial: Helical Processing using EMPIAR-10031 (MAVS)
      • Tutorial: Maximum Box Sizes for Refinement
      • Tutorial: CTF Refinement
      • Tutorial: Ewald Sphere Correction
      • Tutorial: Symmetry Relaxation
      • Tutorial: Orientation Diagnostics
      • Tutorial: BILD files in CryoSPARC v4.4+
      • Tutorial: Mask Creation
      • Case Study: Yeast U4/U6.U5 tri-snRNP
      • Tutorial: 3D Classification
      • Tutorial: 3D Variability Analysis (Part One)
      • Tutorial: 3D Variability Analysis (Part Two)
      • Tutorial: 3D Flexible Refinement
        • Installing 3DFlex Dependencies (v4.1–v4.3)
      • Tutorial: 3D Flex Mesh Preparation
    • Webinar Recordings
  • Real-time processing in cryoSPARC Live
    • About CryoSPARC Live
    • Prerequisites and Compute Resources Setup
    • How to Access cryoSPARC Live
    • UI Overview
    • New Live Session: Start to Finish Guide
    • CryoSPARC Live Tutorial Videos
    • Live Jobs and Session-Level Functions
    • Performance Metrics
    • Managing a CryoSPARC Live Session from the CLI
    • FAQs and Troubleshooting
  • Guides for v3
    • v3 User Interface Guide
      • Dashboard
      • Project and Workspace Management
      • Create and Build Jobs
      • Queue Job, Inspect Job and Other Job Actions
      • View and Download Results
      • Job Relationships
      • Resource Manager
      • User Management
    • Tutorial: Job Builder
    • Get Started with CryoSPARC: Introductory Tutorial (v3)
    • Tutorial: Manually Curate Exposures (v3)
  • Resources
    • Questions and Support
Powered by GitBook
On this page
  • Description
  • Input
  • Common parameters
  • Output
  • Common next steps
  • New in CryoSPARC v4.0+
  • Algorithmic Changes
  • Diagnostic plots
  1. Processing Data in cryoSPARC
  2. All Job Types in CryoSPARC
  3. Conformational Variability

Job: 3D Classification

3D classification without alignment.

PreviousJob: 3D Variability DisplayNextJob: Regroup 3D Classes

Last updated 1 year ago

This job type has been substantially improved from its original release in CryoSPARC v3.3. Changes in v4.0+ (last updated for v4.5) are described below.

Description

3D Classification, first introduced in v3.3, can help discover discrete heterogeneity in single particle cryo-EM datasets. This job currently implements a version of 3D classification without alignment — a classification routine that can complement the and jobs in finding new discrete classes of data.

In CryoSPARC v4.0, 3D Classification was updated with several notable improvements, including FSC regularization, focus and solvent mask inputs, new convergence criteria, and a number of new diagnostic plots and outputs.

Note that in CryoSPARC v4.0+, cloning a 3D classification job that was created in CryoSPARC v3.3 will fail to launch due to a change in the inputs and parameters of the job type. Instead, please create a 3D classification job from scratch in v4.0 and re-connect the desired inputs and set parameters.

Under the hood, 3D Classification uses a combination of Online and Full-Batch Expectation Maximization (O-EM, and F-EM, respectively). These algorithms alternate between (1) computing the most likely class assignments for each particle image in a batch based on known 3D class volumes, and (2) updating each 3D volume based on these assignments.

Please also refer to the , which has been updated for v4.1 with new considerations and example datasets.

Input

  • Particles (with alignments3D)

  • [Optional] Initial Volumes

    • To be used with the input initialization mode. The number of initial volumes must match the number of classes.

  • [Optional] Solvent mask

    • If not supplied, a solvent mask is computed by dilating and soft-padding the consensus volume.

  • [Optional] Focus mask

    • If not supplied, only the solvent mask will be used (i.e., the focus mask will be set to a volume of all ones).

Common parameters

  • Number of classes: Number of classes to use in job. Note that this can be significantly larger than Heterogeneous Refinement for the same computational cost.

  • Filter resolution(parameter name and default updated in v4.5): Classification is performed at this resolution. This parameter must be set for the classification to run. Results are best when the resolution is just high enough to see the difference of interest. For instance:

    • 3-6Å for small changes in density (presence/absence of ligand),

    • 6-10Å for conformational changes between one domain in relation to another,

    • and >10Å for presence/absence of a domain or binding partner.

Prior to v4.5, this parameter is called Target resolution and it has a default of 6 A.

  • Output data after every F-EM iter(updated in v4.1.2): This option may be useful for larger datasets where one may want to monitor the 3D volumes prior to the completion of the job. Note that as of CryoSPARC v4.1.2, this option can only be turned on if class re-ordering is turned off (see below).

  • Symmetry (new in v4.5): Enforce point-group symmetry during back-projection of every class volume.

If 3D Classification is not producing good results, adjusting the following parameters may be a good starting point to get improved results:

  • O-EM learning rate init (default updated in v4.0): For a fixed O-EM batch size and epoch value, larger values will generally result in fewer populated classes

  • Use FSC to filter each class(new in v4.0): FSC filtering may be turned off to match the filtering behaviour of 3D classification in CryoSPARC v3.3.x.

  • Convergence criterion (%) (new in v4.0): Primary stopping criterion — percentage of particles that have switched classes across F-EM iterations. Increasing this value may result in ‘early stopping’ of the optimization.

  • RMS density change convergence check (new in v4.0, default updated in v4.5): If some particles have high probability of being in two or more different classes, the primary switching criterion may result in several F-EM iterations where a substantial number of particles switch classes but the class volumes do not differ significantly. To prevent unnecessary computation, this secondary criterion tracks the root mean square difference of the real-space class volumes across iterations. The job will converge when either criterion is satisfied.

  • Per-particle scale (new in v4.1): Per-particle optimization can be turned off and scales can be set to their upstream values (input) or to a constant value of 1.0 (none).

  • Force hard classification (new in v4.0): Turn off weighted back projection — this may improve performance for small(er) targets where the standard optimization may ‘smear’ a portion of particles across several classes.

Other salient considerations with regards to parameters:

  • Reorder classes by size (new in v4.1.2, default updated in v4.5): With this parameter turned on (default off), classes will be reordered according to their size (i.e., assigned particles) at the end of classification, prior to output generation. To avoid potential confusion regarding class outputs, this option must be turned off if Output data after every F-EM iter is turned on.

Output

  • All particles

  • All volumes

  • Solvent mask (passthrough input or auto-generated)

  • Consensus volume

  • Focus mask (passthrough input if provided)

  • Particles for each class

  • 3D volumes for each class

Common next steps

    • This job can be useful to reconstruct classes at a larger box size than the one set by the 3D classification target resolution.

    • For large sets of classes (e.g., 50+), this job can quickly group these classes into a smaller set of 'superclasses' based on real-space voxel correlations.

  • Further classification of subsets of classes

New in CryoSPARC v4.0+

A number of significant improvements to 3D Classification were added in CryoSPARC v4.0. We list them below.

Algorithmic Changes

  • Per-particle scale optimization (v4.1+)

    • By default, 3D Classification will perform per-particle scale optimization before starting the main EM classification loop.

  • FSC-based filtering (v4.0+)

    • By default, during both O-EM and F-EM iterations, 3D Classification will filter each class volume by its intra-class FSC curve.

  • Convergence criteria (v4.0+)

    • F-EM iterations will conclude when one of two stopping criteria is met:

      • % of particles that switch classes (primary stopping criterion)

      • weighted mean RMS density change falls below a threshold (optional, secondary criterion)

  • Separate focus and solvent mask inputs (v4.0+)

    • 3D Classification accepts two different types of masks. A solvent mask, SSS, and a focus mask, FFF. During optimization we use the following real-space volume for all likelihood computations of class kkk:

Vk←S∗(F∗Vk+(1−F)∗Vˉ),V_k \leftarrow S * (F * V_k + (1-F)*\bar{V}),Vk​←S∗(F∗Vk​+(1−F)∗Vˉ),

where Vˉ\bar{V}Vˉ is the consensus reconstruction.

If F F F is not provided, we set F=1 F = 1 F=1 and apply Vk←S∗VkV_k \leftarrow S * V_kVk​←S∗Vk​. Otherwise, we also plot real-space slices and projections of the mask overlayed on the consensus volume map:

  • Filtered consensus volume output (v4.4+)

    • The consensus map is now filtered in accordance to its FSC. The resulting map is output by the job for inspection.

Diagnostic plots

Starting with CryoSPARC v4.0, 3D Classification outputs several new diagnostic plots listed below.

Per-particle Class ESS Histogram (added in v4.0)

This histogram can help diagnose poor classification results by showing if some particles have significant probability mass in more than one class. The ESS (Effective Sample Size) is a measure of how many classes each particle appears to belong to with significant probability. And ESS of 1.0 indicates that a particle is completely confidently assigned to only one class. An ESS of 2.0 would mean that a particle belongs with substantial probability to two classes. When many particles have a large ESS (> 1), this indicates that there is significant uncertainty in classification, any the classes may be overlapping or similar.

Difference from Consensus Real-Space Slices (added in v3.3, updated in v4.0)

This plot shows the real-space difference between the consensus map and each class map, regularized by the class FSC (if FSC regularization is turned on). This can quickly show areas of heterogeneity.

Class Flow Diagram (added in v4.0, updated in v4.1)

This diagram shows how many particles switched classes across F-EM iterations (output starts at the second F-EM iteration). An edge, (i,j), is drawn with a thickness, colour, and opacity defined by the amount of particles that switch from class i to class j.

Class Flow Matrix (added in v4.1)

This diagram visualizes class flow in a matrix format. Each column represents a 1D distribution of the particles in a given class at the current F-EM iteration. Each row represents the class which the particles belonged to at the previous iteration. In other words, each square in this grid represents an edge in the bipartite class flow graph above. This form of class flow can be useful in visualizing 'minor' edges that are difficult to see in the bipartite graph, and it can greatly improve clarity for class flow with large (25+) numbers of classes.

Class Assignment Histogram (added in v3.3, updated in v4.0)

This histogram now includes both total assignments and the ‘effective size’ of the class. The latter is a sum of the probability mass in that class. When the assignments and effective size bars are differently sized, this indicates that there is uncertainty in the classification, as many particles have probabilities that are spread out between classes (an effect included in the effective size) compared to the class where they have the maximum probability (the assignments).

This parameter will control the box size and pixel size of the class volumes. To reconstruct classes at their extracted box size, use the .

This output is new in CryoSPARC v4.5+ and is a volumes group output. It includes a series result that contains a downloadable zip file of all volumes. See for more details.

Heterogeneous Refinement
3D Variability
3D Classification tutorial
Heterogeneous Reconstruction Only job
documentation for volumes groups
Job: Heterogeneous Reconstruction Only
Job: Regroup 3D
Focus mask overlayed on real-space slices.