CryoSPARC Guide
  • About CryoSPARC
  • Current Version
  • Licensing
    • Non-commercial license agreement
  • Setup, Configuration and Management
    • CryoSPARC Architecture and System Requirements
    • CryoSPARC Installation Prerequisites
    • How to Download, Install and Configure
      • Obtaining A License ID
      • Downloading and Installing CryoSPARC
      • CryoSPARC Cluster Integration Script Examples
      • Accessing the CryoSPARC User Interface
    • Deploying CryoSPARC on AWS
      • Performance Benchmarks
    • Using CryoSPARC with Cluster Management Software
    • Software Updates and Patches
    • Management and Monitoring
      • Environment variables
      • (Optional) Hosting CryoSPARC Through a Reverse Proxy
      • cryosparcm reference
      • cryosparcm cli reference
      • cryosparcw reference
    • Software System Guides
      • Guide: Updating to CryoSPARC v4
      • Guide: Installation Testing with cryosparcm test
      • Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)
      • Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)
      • Guide: Performance Benchmarking (v4.3+)
      • Guide: Download Error Reports
      • Guide: Maintenance Mode and Configurable User Facing Messages
      • Guide: User Management
      • Guide: Multi-user Unix Permissions and Data Access Control
      • Guide: Lane Assignments and Restrictions
      • Guide: Queuing Directly to a GPU
      • Guide: Priority Job Queuing
      • Guide: Configuring Custom Variables for Cluster Job Submission Scripts
      • Guide: SSD Particle Caching in CryoSPARC
      • Guide: Data Management in CryoSPARC (v4.0+)
      • Guide: Data Cleanup (v4.3+)
      • Guide: Reduce Database Size (v4.3+)
      • Guide: Data Management in CryoSPARC (≤v3.3)
      • Guide: CryoSPARC Live Session Data Management
      • Guide: Manipulating .cs Files Created By CryoSPARC
      • Guide: Migrating your CryoSPARC Instance
      • Guide: EMDB-friendly XML file for FSC plots
    • Troubleshooting
  • Application Guide (v4.0+)
    • A Tour of the CryoSPARC Interface
    • Browsing the CryoSPARC Instance
    • Projects, Workspaces and Live Sessions
    • Jobs
    • Job Views: Cards, Tree, and Table
    • Creating and Running Jobs
    • Low Level Results Interface
    • Filters and Sorting
    • View Options
    • Tags
    • Flat vs Hierarchical Navigation
    • File Browser
    • Blueprints
    • Workflows
    • Inspecting Data
    • Managing Jobs
    • Interactive Jobs
    • Upload Local Files
    • Managing Data
    • Downloading and Exporting Data
    • Instance Management
    • Admin Panel
  • Cryo-EM Foundations
    • Image Formation
      • Contrast in Cryo-EM
      • Waves as Vectors
      • Aliasing
  • Expectation Maximization in Cryo-EM
  • Processing Data in cryoSPARC
    • Get Started with CryoSPARC: Introductory Tutorial (v4.0+)
    • Tutorial Videos
    • All Job Types in CryoSPARC
      • Import
        • Job: Import Movies
        • Job: Import Micrographs
        • Job: Import Particle Stack
        • Job: Import 3D Volumes
        • Job: Import Templates
        • Job: Import Result Group
        • Job: Import Beam Shift
      • Motion Correction
        • Job: Patch Motion Correction
        • Job: Full-Frame Motion Correction
        • Job: Local Motion Correction
        • Job: MotionCor2 (Wrapper) (BETA)
        • Job: Reference Based Motion Correction (BETA)
      • CTF Estimation
        • Job: Patch CTF Estimation
        • Job: Patch CTF Extraction
        • Job: CTFFIND4 (Wrapper)
        • Job: Gctf (Wrapper) (Legacy)
      • Exposure Curation
        • Job: Micrograph Denoiser (BETA)
        • Job: Micrograph Junk Detector (BETA)
        • Interactive Job: Manually Curate Exposures
      • Particle Picking
        • Interactive Job: Manual Picker
        • Job: Blob Picker
        • Job: Template Picker
        • Job: Filament Tracer
        • Job: Blob Picker Tuner
        • Interactive Job: Inspect Particle Picks
        • Job: Create Templates
      • Extraction
        • Job: Extract from Micrographs
        • Job: Downsample Particles
        • Job: Restack Particles
      • Deep Picking
        • Guideline for Supervised Particle Picking using Deep Learning Models
        • Deep Network Particle Picker
          • T20S Proteasome: Deep Particle Picking Tutorial
          • Job: Deep Picker Train and Job: Deep Picker Inference
        • Topaz (Bepler, et al)
          • T20S Proteasome: Topaz Particle Picking Tutorial
          • T20S Proteasome: Topaz Micrograph Denoising Tutorial
          • Job: Topaz Train and Job: Topaz Cross Validation
          • Job: Topaz Extract
          • Job: Topaz Denoise
      • Particle Curation
        • Job: 2D Classification
        • Interactive Job: Select 2D Classes
        • Job: Reference Based Auto Select 2D (BETA)
        • Job: Reconstruct 2D Classes
        • Job: Rebalance 2D Classes
        • Job: Class Probability Filter (Legacy)
        • Job: Rebalance Orientations
        • Job: Subset Particles by Statistic
      • 3D Reconstruction
        • Job: Ab-Initio Reconstruction
      • 3D Refinement
        • Job: Homogeneous Refinement
        • Job: Heterogeneous Refinement
        • Job: Non-Uniform Refinement
        • Job: Homogeneous Reconstruction Only
        • Job: Heterogeneous Reconstruction Only
        • Job: Homogeneous Refinement (Legacy)
        • Job: Non-uniform Refinement (Legacy)
      • CTF Refinement
        • Job: Global CTF Refinement
        • Job: Local CTF Refinement
        • Job: Exposure Group Utilities
      • Conformational Variability
        • Job: 3D Variability
        • Job: 3D Variability Display
        • Job: 3D Classification
        • Job: Regroup 3D Classes
        • Job: Reference Based Auto Select 3D (BETA)
        • Job: 3D Flexible Refinement (3DFlex) (BETA)
      • Postprocessing
        • Job: Sharpening Tools
        • Job: DeepEMhancer (Wrapper)
        • Job: Validation (FSC)
        • Job: Local Resolution Estimation
        • Job: Local Filtering
        • Job: ResLog Analysis
        • Job: ThreeDFSC (Wrapper) (Legacy)
      • Local Refinement
        • Job: Local Refinement
        • Job: Particle Subtraction
        • Job: Local Refinement (Legacy)
      • Helical Reconstruction
        • Helical symmetry in CryoSPARC
        • Job: Helical Refinement
        • Job: Symmetry search utility
        • Job: Average Power Spectra
      • Utilities
        • Job: Exposure Sets Tool
        • Job: Exposure Tools
        • Job: Generate Micrograph Thumbnails
        • Job: Cache Particles on SSD
        • Job: Check for Corrupt Particles
        • Job: Particle Sets Tool
        • Job: Reassign Particles to Micrographs
        • Job: Remove Duplicate Particles
        • Job: Symmetry Expansion
        • Job: Volume Tools
        • Job: Volume Alignment Tools
        • Job: Align 3D maps
        • Job: Split Volumes Group
        • Job: Orientation Diagnostics
      • Simulations
        • Job: Simulate Data (GPU)
        • Job: Simulate Data (Legacy)
    • CryoSPARC Tools
    • Data Processing Tutorials
      • Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)
      • Case Study: DkTx-bound TRPV1 (EMPIAR-10059)
      • Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)
      • Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)
      • Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)
      • Case Study: Exploratory data processing by Oliver Clarke
      • Tutorial: Tips for Membrane Protein Structures
      • Tutorial: Common CryoSPARC Plots
      • Tutorial: Negative Stain Data
      • Tutorial: Phase Plate Data
      • Tutorial: EER File Support
      • Tutorial: EPU AFIS Beam Shift Import
      • Tutorial: Patch Motion and Patch CTF
      • Tutorial: Float16 Support
      • Tutorial: Particle Picking Calibration
      • Tutorial: Blob Picker Tuner
      • Tutorial: Helical Processing using EMPIAR-10031 (MAVS)
      • Tutorial: Maximum Box Sizes for Refinement
      • Tutorial: CTF Refinement
      • Tutorial: Ewald Sphere Correction
      • Tutorial: Symmetry Relaxation
      • Tutorial: Orientation Diagnostics
      • Tutorial: BILD files in CryoSPARC v4.4+
      • Tutorial: Mask Creation
      • Case Study: Yeast U4/U6.U5 tri-snRNP
      • Tutorial: 3D Classification
      • Tutorial: 3D Variability Analysis (Part One)
      • Tutorial: 3D Variability Analysis (Part Two)
      • Tutorial: 3D Flexible Refinement
        • Installing 3DFlex Dependencies (v4.1–v4.3)
      • Tutorial: 3D Flex Mesh Preparation
    • Webinar Recordings
  • Real-time processing in cryoSPARC Live
    • About CryoSPARC Live
    • Prerequisites and Compute Resources Setup
    • How to Access cryoSPARC Live
    • UI Overview
    • New Live Session: Start to Finish Guide
    • CryoSPARC Live Tutorial Videos
    • Live Jobs and Session-Level Functions
    • Performance Metrics
    • Managing a CryoSPARC Live Session from the CLI
    • FAQs and Troubleshooting
  • Guides for v3
    • v3 User Interface Guide
      • Dashboard
      • Project and Workspace Management
      • Create and Build Jobs
      • Queue Job, Inspect Job and Other Job Actions
      • View and Download Results
      • Job Relationships
      • Resource Manager
      • User Management
    • Tutorial: Job Builder
    • Get Started with CryoSPARC: Introductory Tutorial (v3)
    • Tutorial: Manually Curate Exposures (v3)
  • Resources
    • Questions and Support
Powered by GitBook
On this page
  • Introduction
  • Prerequisites
  • Creating and Running the Extensive Workflow
  • Troubleshooting Failed Jobs
  • Additional Extensive Testing
  • Expected Results
  1. Setup, Configuration and Management
  2. Software System Guides

Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)

PreviousGuide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)NextGuide: Performance Benchmarking (v4.3+)

Last updated 1 year ago

For the version of this guide applicable to CryoSPARC versions v4.3.0+, please see:

Introduction

CryoSPARC provides a job called "Extensive Workflow for T20S", which performs a full 3D reconstruction of the T20S Proteasome () from a small (~8GB) subset of movies. The CryoSPARC engineering team uses this job to automatically test and benchmark CryoSPARC between releases.

System Administrators may use the extensive workflow job to verify that CryoSPARC is correctly configured following a fresh installation or an update.

  • Import Micrographs

  • Motion Correction

  • CTF Estimation

  • Particle Picking and Extraction

  • 2D Classification

  • Ab-initio reconstruction

  • Homogeneous Refinement

The following system requirements are verified:

  • CryoSPARC system and license installation

  • Worker/Cluster configuration

  • GPU and CUDA driver installation

  • SSD caching

The sample data has the following characteristics:

  • Number of images: 20

  • Frames per image: 38

  • Image size: 7420 × 7676 (K2 Super Resolution)

  • Pixel size: 0.66 Å

Once started, the workflow should take no more than an hour to complete.

Prerequisites

Creating and Running the Extensive Workflow

  1. Open the CryoSPARC web interface

  2. In the dashboard, create a new Project from the navigation bar

Specify a descriptive title such as "Extensive Workflow Testing" and directory for the project to store its data.

3. Create a new workspace for that project.

Best practices: Create a new workspace and run the Extensive Workflow in that workspace each time CryoSPARC updates and restarts. Name each workspace with the latest installed version of CryoSPARC that the job runs on. For example, when testing CryoSPARC v2.15.0, name the workspace "v2.15.0 Benchmark & Validation"

4. Select the Job Builder from the sidebar and select the "Extensive Workflow for T20S (BENCH) (BETA)" job (under Workflows)

5. (Optional) If desired, change the workflow parameters.

6. Select "Queue" and choose a worker lane for the job, then select "Create"

After queuing, a modal opens with an overview of the workflow job progress. The job status should shortly change to "Running".

Close the modal with the × button. This shows a workspace overview with the child jobs that the Extensive Workflow job spawns to carry out T20S processing.

Once all child jobs successfully complete, the Extensive Workflow job status changes to "Completed". This means the installation was successful. Users may now be notified to start or resume processing!

Troubleshooting Failed Jobs

If any child jobs fails, the extensive workflow times-out and its status is set to "Failed".

Scroll through the workspace to find other jobs with the "Failed" status. Open the job overview either by selecting the job number next to the status indicator (e.g., J4) , or by selecting the Job card and pressing the Space key.

Scroll to the bottom to see why the job failed.

Example Import Movies failure because the Gain Reference was not found

Common failure reasons include

Once the configuration issue is resolved, restart the Extensive Workflow job: Either create a new workspace and job as already noted, or clear the existing Extensive Workflow job and re-queue.

Additional Extensive Testing

For an even more extensive test of robustness, the Extensive Workflow job provides an advanced option called "Run all job types"

  1. Enable advanced mode near the top of the job builder

  2. Enable to "Run all job types" switch

With this option enabled, the job runs additional child jobs in parallel. Use this to verify multi-GPU performance on a single node.

The following additional job types are included:

  • Full-frame motion correction

  • Global CTF estimation

  • Local motion correction

  • Multi-class ab-initio reconstruction

  • Heterogeneous and non-uniform refinement

  • 3D Variability

Expected Results

Below are the results from our tests with CryoSPARC v3.1 on a 4GPU machine with the T20S subset.

Job Type

Approximate Run Time (seconds)

Import Movies

92

Patch Motion Correction (Multi)

220

Full Frame Motion Correction (Multi)

75

Patch CTF Estimation (Multi)

66

Curate Exposures

1.1

Blob Picker

12

Template Picker

13

Inspect Picks

12

Extract from Micrographs (CPU)

39

Extract from Micrographs (GPU)

43

Local Motion Correction

180

2D Classification

280

Select 2D Classes

7.5

Ab-Initio Reconstruction (1 class)

450

Ab-Initio Reconstruction (3 class)

800

Homogenous Refinement

1940

Heterogeneous Refinement (3 class)

3000

Non-Uniform Refinement

4300

Sharpen

32

Validation

94

Global CTF Refinement

41

Local CTF Refinement

46

3D Variability

560

3D Variability Display

140

The Extensive Workflow covers the , including the following workflow jobs:

Specifying a valid "Movies data path" and "Gain reference path" is NOT required; if the path does not exist on the system, cryoSPARC automatically downloads a ~8GB subset of the and deletes the download when the job finishes.

or license key entered incorrectly

On a machine with the , the Extensive Workflow takes ~1 hour with the default settings and ~1 hour 30 minutes with all job types enabled (note that some jobs run in parallel when enough GPUs are available).

full T20S tutorial
How to Download, Install and Configure
T20S dataset
Cannot verify license
Incorrect filesystem permissions
CUDA not set up correctly
SSD cache full or not set up correctly
Worker registration issue
Not enough GPU memory available
Extensive Validation
EMPIAR-10025
recommended system configuration