CryoSPARC Guide
  • About CryoSPARC
  • Current Version
  • Licensing
    • Non-commercial license agreement
  • Setup, Configuration and Management
    • CryoSPARC Architecture and System Requirements
    • CryoSPARC Installation Prerequisites
    • How to Download, Install and Configure
      • Obtaining A License ID
      • Downloading and Installing CryoSPARC
      • CryoSPARC Cluster Integration Script Examples
      • Accessing the CryoSPARC User Interface
    • Deploying CryoSPARC on AWS
      • Performance Benchmarks
    • Using CryoSPARC with Cluster Management Software
    • Software Updates and Patches
    • Management and Monitoring
      • Environment variables
      • (Optional) Hosting CryoSPARC Through a Reverse Proxy
      • cryosparcm reference
      • cryosparcm cli reference
      • cryosparcw reference
    • Software System Guides
      • Guide: Updating to CryoSPARC v4
      • Guide: Installation Testing with cryosparcm test
      • Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)
      • Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)
      • Guide: Performance Benchmarking (v4.3+)
      • Guide: Download Error Reports
      • Guide: Maintenance Mode and Configurable User Facing Messages
      • Guide: User Management
      • Guide: Multi-user Unix Permissions and Data Access Control
      • Guide: Lane Assignments and Restrictions
      • Guide: Queuing Directly to a GPU
      • Guide: Priority Job Queuing
      • Guide: Configuring Custom Variables for Cluster Job Submission Scripts
      • Guide: SSD Particle Caching in CryoSPARC
      • Guide: Data Management in CryoSPARC (v4.0+)
      • Guide: Data Cleanup (v4.3+)
      • Guide: Reduce Database Size (v4.3+)
      • Guide: Data Management in CryoSPARC (≤v3.3)
      • Guide: CryoSPARC Live Session Data Management
      • Guide: Manipulating .cs Files Created By CryoSPARC
      • Guide: Migrating your CryoSPARC Instance
      • Guide: EMDB-friendly XML file for FSC plots
    • Troubleshooting
  • Application Guide (v4.0+)
    • A Tour of the CryoSPARC Interface
    • Browsing the CryoSPARC Instance
    • Projects, Workspaces and Live Sessions
    • Jobs
    • Job Views: Cards, Tree, and Table
    • Creating and Running Jobs
    • Low Level Results Interface
    • Filters and Sorting
    • View Options
    • Tags
    • Flat vs Hierarchical Navigation
    • File Browser
    • Blueprints
    • Workflows
    • Inspecting Data
    • Managing Jobs
    • Interactive Jobs
    • Upload Local Files
    • Managing Data
    • Downloading and Exporting Data
    • Instance Management
    • Admin Panel
  • Cryo-EM Foundations
    • Image Formation
      • Contrast in Cryo-EM
      • Waves as Vectors
      • Aliasing
  • Expectation Maximization in Cryo-EM
  • Processing Data in cryoSPARC
    • Get Started with CryoSPARC: Introductory Tutorial (v4.0+)
    • Tutorial Videos
    • All Job Types in CryoSPARC
      • Import
        • Job: Import Movies
        • Job: Import Micrographs
        • Job: Import Particle Stack
        • Job: Import 3D Volumes
        • Job: Import Templates
        • Job: Import Result Group
        • Job: Import Beam Shift
      • Motion Correction
        • Job: Patch Motion Correction
        • Job: Full-Frame Motion Correction
        • Job: Local Motion Correction
        • Job: MotionCor2 (Wrapper) (BETA)
        • Job: Reference Based Motion Correction (BETA)
      • CTF Estimation
        • Job: Patch CTF Estimation
        • Job: Patch CTF Extraction
        • Job: CTFFIND4 (Wrapper)
        • Job: Gctf (Wrapper) (Legacy)
      • Exposure Curation
        • Job: Micrograph Denoiser (BETA)
        • Job: Micrograph Junk Detector (BETA)
        • Interactive Job: Manually Curate Exposures
      • Particle Picking
        • Interactive Job: Manual Picker
        • Job: Blob Picker
        • Job: Template Picker
        • Job: Filament Tracer
        • Job: Blob Picker Tuner
        • Interactive Job: Inspect Particle Picks
        • Job: Create Templates
      • Extraction
        • Job: Extract from Micrographs
        • Job: Downsample Particles
        • Job: Restack Particles
      • Deep Picking
        • Guideline for Supervised Particle Picking using Deep Learning Models
        • Deep Network Particle Picker
          • T20S Proteasome: Deep Particle Picking Tutorial
          • Job: Deep Picker Train and Job: Deep Picker Inference
        • Topaz (Bepler, et al)
          • T20S Proteasome: Topaz Particle Picking Tutorial
          • T20S Proteasome: Topaz Micrograph Denoising Tutorial
          • Job: Topaz Train and Job: Topaz Cross Validation
          • Job: Topaz Extract
          • Job: Topaz Denoise
      • Particle Curation
        • Job: 2D Classification
        • Interactive Job: Select 2D Classes
        • Job: Reference Based Auto Select 2D (BETA)
        • Job: Reconstruct 2D Classes
        • Job: Rebalance 2D Classes
        • Job: Class Probability Filter (Legacy)
        • Job: Rebalance Orientations
        • Job: Subset Particles by Statistic
      • 3D Reconstruction
        • Job: Ab-Initio Reconstruction
      • 3D Refinement
        • Job: Homogeneous Refinement
        • Job: Heterogeneous Refinement
        • Job: Non-Uniform Refinement
        • Job: Homogeneous Reconstruction Only
        • Job: Heterogeneous Reconstruction Only
        • Job: Homogeneous Refinement (Legacy)
        • Job: Non-uniform Refinement (Legacy)
      • CTF Refinement
        • Job: Global CTF Refinement
        • Job: Local CTF Refinement
        • Job: Exposure Group Utilities
      • Conformational Variability
        • Job: 3D Variability
        • Job: 3D Variability Display
        • Job: 3D Classification
        • Job: Regroup 3D Classes
        • Job: Reference Based Auto Select 3D (BETA)
        • Job: 3D Flexible Refinement (3DFlex) (BETA)
      • Postprocessing
        • Job: Sharpening Tools
        • Job: DeepEMhancer (Wrapper)
        • Job: Validation (FSC)
        • Job: Local Resolution Estimation
        • Job: Local Filtering
        • Job: ResLog Analysis
        • Job: ThreeDFSC (Wrapper) (Legacy)
      • Local Refinement
        • Job: Local Refinement
        • Job: Particle Subtraction
        • Job: Local Refinement (Legacy)
      • Helical Reconstruction
        • Helical symmetry in CryoSPARC
        • Job: Helical Refinement
        • Job: Symmetry search utility
        • Job: Average Power Spectra
      • Utilities
        • Job: Exposure Sets Tool
        • Job: Exposure Tools
        • Job: Generate Micrograph Thumbnails
        • Job: Cache Particles on SSD
        • Job: Check for Corrupt Particles
        • Job: Particle Sets Tool
        • Job: Reassign Particles to Micrographs
        • Job: Remove Duplicate Particles
        • Job: Symmetry Expansion
        • Job: Volume Tools
        • Job: Volume Alignment Tools
        • Job: Align 3D maps
        • Job: Split Volumes Group
        • Job: Orientation Diagnostics
      • Simulations
        • Job: Simulate Data (GPU)
        • Job: Simulate Data (Legacy)
    • CryoSPARC Tools
    • Data Processing Tutorials
      • Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)
      • Case Study: DkTx-bound TRPV1 (EMPIAR-10059)
      • Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)
      • Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)
      • Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)
      • Case Study: Exploratory data processing by Oliver Clarke
      • Tutorial: Tips for Membrane Protein Structures
      • Tutorial: Common CryoSPARC Plots
      • Tutorial: Negative Stain Data
      • Tutorial: Phase Plate Data
      • Tutorial: EER File Support
      • Tutorial: EPU AFIS Beam Shift Import
      • Tutorial: Patch Motion and Patch CTF
      • Tutorial: Float16 Support
      • Tutorial: Particle Picking Calibration
      • Tutorial: Blob Picker Tuner
      • Tutorial: Helical Processing using EMPIAR-10031 (MAVS)
      • Tutorial: Maximum Box Sizes for Refinement
      • Tutorial: CTF Refinement
      • Tutorial: Ewald Sphere Correction
      • Tutorial: Symmetry Relaxation
      • Tutorial: Orientation Diagnostics
      • Tutorial: BILD files in CryoSPARC v4.4+
      • Tutorial: Mask Creation
      • Case Study: Yeast U4/U6.U5 tri-snRNP
      • Tutorial: 3D Classification
      • Tutorial: 3D Variability Analysis (Part One)
      • Tutorial: 3D Variability Analysis (Part Two)
      • Tutorial: 3D Flexible Refinement
        • Installing 3DFlex Dependencies (v4.1–v4.3)
      • Tutorial: 3D Flex Mesh Preparation
    • Webinar Recordings
  • Real-time processing in cryoSPARC Live
    • About CryoSPARC Live
    • Prerequisites and Compute Resources Setup
    • How to Access cryoSPARC Live
    • UI Overview
    • New Live Session: Start to Finish Guide
    • CryoSPARC Live Tutorial Videos
    • Live Jobs and Session-Level Functions
    • Performance Metrics
    • Managing a CryoSPARC Live Session from the CLI
    • FAQs and Troubleshooting
  • Guides for v3
    • v3 User Interface Guide
      • Dashboard
      • Project and Workspace Management
      • Create and Build Jobs
      • Queue Job, Inspect Job and Other Job Actions
      • View and Download Results
      • Job Relationships
      • Resource Manager
      • User Management
    • Tutorial: Job Builder
    • Get Started with CryoSPARC: Introductory Tutorial (v3)
    • Tutorial: Manually Curate Exposures (v3)
  • Resources
    • Questions and Support
Powered by GitBook
On this page
  • CryoSPARC Live Architecture and Reprocessing
  • Live session terminology
  • Job types unique to cryoSPARC Live
  • Session-Level Functions
  1. Real-time processing in cryoSPARC Live

Live Jobs and Session-Level Functions

Overview of CryoSPARC Live architecture and job types specific to Live.

CryoSPARC Live Architecture and Reprocessing

CryoSPARC Live is built on the idea that one should be able to experiment with parameters on the fly and that the software should be able to efficiently reprocess/redo the steps that are necessary in order to test or effect parameter changes, while maintaining the overall progress of the Live session.

CryoSPARC Live tracks and manages exposures, particles, results, volumes, etc in a new architecture that allows for consistent management of session state, reprocessing only the necessary work when parameters are changed, tracking acceptance/rejection of images, and live streaming of particles through various stages of processing. This new underlying system supports many new features, and also requires some job types and session-level functions unique to CryoSPARC Live (as compared with the main CryoSPARC application).

Live session terminology

  • Session: The processing workflow for doing real-time preprocessing and 2D/3D processing in CryoSPARC Live. New Sessions ("SX") can be configured from the CryoSPARC Live Application directly and they are housed within an existing CryoSPARC Project. Live Sessions are the equivalent of Workspaces ("WX") in the regular CryoSPARC Application. Therefore, there each session maps to one workspace.

  • Exposure: An individual image file that CryoSPARC Live preprocesses. The "Exposure" terminology is used to refer to both movies and micrographs interchangeably.

  • Exposure Group: A collection of exposures with the same optical parameters. Mainly includes information about where to find the exposures on disk, and how CryoSPARC can find them. A Live Session can have multiple exposure groups, which can be used to separate data collected from different data collection sessions, grids, or beam tilt groups. Exposures and their extracted particles can be ignored from downstream processing by ignoring the exposure groups from which they came via the Configuration tab.

Job types unique to cryoSPARC Live

  • CryoSPARC Live Session Job: The job created when a Live Session is created. This job, visible in the CryoSPARC interface, is the parent to all jobs later created by a running Live Session.

  • CryoSPARC Live Worker: The live preprocessing worker that executes motion correction, CTF estimation, thumbnail generation, particle picking and particle extraction for each exposure it encounters.

  • Streaming 2D Classification: An adapted version of 2D Classification that can classify new particles as they are extracted by the CryoSPARC Live Worker in a streaming fashion.

  • Streaming Refinement: An adapted version of Homogeneous Refinement that can refine a volume with new particles as they are classified by the Streaming 2D Classification job in a streaming fashion.

  • CryoSPARC Live Exposure Export: Exports all valid exposures from a Live Session to its CryoSPARC workspace, for use in further processing.

  • CryoSPARC Live Particle Export: Exports all valid particles from a Live Session to its CryoSPARC workspace, for use in further processing.

Session-Level Functions

There are a few high-level Session Functions that can be performed.

Start Session: This will cause the Preprocessing GPU Worker(s) to start reading in movies and perform preprocessing steps (motion correction, CTF estimation, particle picking and extraction). You must have filled out and saved all required parameters in the Configuration Tab in order to Start a Session.

Pause Session: This will cause any currently running and queued CryoSPARC Live jobs (Preprocessing GPU Workers(s), Streaming 2D Classification, Ab-Initio Reconstruction and Streaming Refinement) to be killed and marked as completed. Once paused, the session can be started again to continue processing.

In order to edit any Configuration or Compute Resources parameters from the Configuration Tab, e.g., changing the Number of Preprocessing GPU Workers during a running session, you will need to Pause Session and then Start Session again for them to take effect.

Mark as Completed: This will set the status of the session to "Completed". Doing this allows you to organize your sessions, as well as manage the data created by the session via the "Manage Data" tab.

Clear Session: This will delete all data created by the session, and remove all intermediate results created by the session. Only the parameters that were used to set up the session will remain, to allow for easy re-starting of the session.

Modify Exposure Processing Priority: There are four modes available that modify which exposures are prioritized to be processed by the Preprocessing GPU Worker(s). This is available as of CryoSPARC v3.3.2+220824 and later.

  • normal: Exposures are processed in ascending UID order, unless exposures that need to be reprocessed are available. In that case, exposures will be processed in descending UID order. This is the default priority mode.

  • oldest: The oldest found exposure that needs to be processed (or reprocessed) will be prioritized.

  • latest: The latest found exposure that needs to be processed (or reprocessed) will be prioritized.

  • alternate: Alternate between oldest and latest priority modes.

Test exposures and exposures manually set to be reprocessed will always be processed first, no matter the priority mode.

The exposure processing priority of a session can be modified at any time. To modify it, use cryosparcm rtpcli to run the following command:

cryosparcm rtpcli "set_session_exposure_processing_priority('<project_uid>', '<session_uid>', '<priority_value>')"

For example: cryosparcm rtpcli "set_session_exposure_processing_priority('P3', 'S1', 'latest')"

PreviousCryoSPARC Live Tutorial VideosNextPerformance Metrics

Last updated 1 year ago