CryoSPARC Architecture and System Requirements

CryoSPARC System Architecture Overview

CryoSPARC is a backend and frontend high-performance computing software system that provides data processing and image analysis capabilities for single particle cryo-EM, along with a rich browser-based user interface and command line tools.

CryoSPARC can be deployed on-premises or in the cloud.

Master-worker pattern

The system is based on a master-worker pattern.

• The master processes (web application, core application and MongoDB database) run together on one machine (master node). The master node requires relatively lightweight resources (4+ CPUs, 16GB+ RAM, 250GB+ HDD storage)

• Worker processes run on any available/configured machine that has NVIDIA GPUs (worker node). The worker is responsible for all actual computation and data handling and is dispatched by the master node.

The master-worker architecture allows CryoSPARC to be installed and scaled up flexibly on a variety of hardware, including a single workstation, groups of workstations, cluster nodes, HPC clusters, cloud nodes, and more.

Typical CryoSPARC System Setups

Single Workstation

Both the CryoSPARC master and CryoSPARC worker processes may run on the same machine. The only requirement is that GPU resources are available for the CryoSPARC worker processes. This is the simplest setup.

Master-Worker

In the master-worker setup, the CryoSPARC master is installed on a lightweight machine, and the worker processes are installed on one or more GPU servers. This is the most flexible setup for installing CryoSPARC. There are three main requirements for this setup, which are also explained in greater detail in the installation sections of this document:

Clusters

The master node can also spawn or submit jobs to a cluster scheduler system (e.g., Slurm Workload Manager). This integration is transparent, and works similar to the master-worker setup explained above, except all resource scheduling is handled by the cluster scheduler, and CryoSPARC's scheduler is only used for orchestration and management of jobs. Similar requirements are present:

Supported cluster schedulers

CryoSPARC supports most cluster schedulers, including SLURM, SGE and PBS. Please see here for example cluster configurations for popular schedulers.

CryoSPARC System Requirements

The following are requirements for every master and worker node in the system unless otherwise specified.

CryoSPARC Master Node Requirements

The following are requirements specific to the master node.

A 10Gbps connection is recommended to the storage servers given raw cryo-EM movies can be several TB in size, and I/O bottlenecks are more of a concern than processing power for pre-processing jobs in CryoSPARC.

Although a CPU with a higher core count is recommended, a CPU with a faster clock rate is more advantageous due to how master processes are implemented.

Enough System Storage is required to host the cryosparc_master installation package and database folder. Each CryoSPARC project occupies between 100MB and 5GB of database storage, depending on the size of the project. 500GB is enough for approximately 200 medium-sized projects. Note that this excludes the space required for CryoSPARC project data in bulk storage, which could be in terabytes for larger projects.

Worker Node/Cluster Worker Minimum Requirements

The following are requirements for each worker node/cluster worker.

High CPU memory bandwidth is especially important for CryoSPARC Live preprocessing.

System RAM is very important for worker nodes and should scale proportionately to the number of GPUs available for processing on the system.

Enough System Storage is required to host the cryosparc_worker installation package.

Fast local storage is also necessary as reconstruction jobs require random access to particle images. SSDs provide high throughput in this context. See the section on Solid State Storage for more details.

Operating System

We recommend running CryoSPARC on an up-to-date long-term support Linux distribution, such as Ubuntu (22.04, 24.04), Rocky Linux (8, 9) or a related distribution.

Disks and compression

Fast disks are a necessity for processing cryo-EM data efficiently. Fast sequential read/write throughput is needed during pre-processing stages (e.g., motion correction) where the volume of data is very large (tens of TB) while the amount of computation is relatively low (sequential processing for motion correction, CTF estimation, particle picking, etc.)

Spinning disk arrays in a RAID configuration are used to store large raw data files, and often cluster file systems are used for larger systems. As a rule of thumb, to saturate a 4-GPU machine during pre-processing, a sustained sequential read of 1000MB/s is required.

Compression can greatly reduce the amount of data stored in movie files, and also greatly speeds up preprocessing because decompression is actually faster than reading uncompressed data straight from disk. Typically, counting-mode movie files are stored in LZW compressed TIFF format without gain correction, so that the gain reference file is stored separately and must be applied on-the-fly during processing (which is supported by CryoSPARC). Compressing gain corrected movies can often result in much worse compression ratios than compressing pre-gain corrected (integer count) data.

CryoSPARC supports LZW compressed TIFF format, EER format and BZ2 compressed MRC format natively. In either case, the gain reference must be supplied as an MRC file. TIFF, EER and BZ2 compression are implemented as multi-core decompression streams on-the-fly.

Solid State Storage (SSDs)

SSD space is optional on a per-worker node basis but is highly recommended for worker nodes that will be running refinements and reconstructions using particle images. Nodes reserved for pre-processing (motion correction, particle picking, CTF estimation, etc) do not need to have an SSD.

CryoSPARC particle processing algorithms rely on random-access patterns and multiple passes through the data, rather than sequentially reading the data at once. Using a storage medium that allows for fast random reads will speed up processing dramatically.

CryoSPARC manages the SSD cache on each worker node transparently. Files are automatically cached, re-used across the same project and deleted if more space is needed. Please see the SSD Caching guide for more information.

The size of your typical single particle cryo-EM datasets will inform the size of SSD you choose to use. For a sample calculation, see:

Guide: SSD Particle Caching in CryoSPARCCryoSPARC Guide

Graphical Processing Units (GPUs)

At least one worker node must have GPUs available to run the complete set of CryoSPARC jobs. Non-GPU workers may run CPU-only jobs.

The GPU memory (VRAM) in each GPU limits the maximum particle box size for reconstruction. Typically, a GPU with 12GB VRAM can handle a box size of up to 700^3, and up to 1024^3 in some job types. See the following tutorial for more details.

Please ensure each connected worker includes a recent version of the Nvidia Driver compatible with your GPU. See this section for details. Download the latest driver for your GPUs here. Visit Troubleshooting to resolve common GPU errors.

Selecting GPUs

When acquiring GPUs to use with CryoSPARC, the following considerations may be useful.

• CryoSPARC almost exclusively uses single-precision operations on the GPU. As such, consumer cards generally have a much better price/performance ratio than enterprise cards. Enterprise cards do have their own benefits such as reliability, better cooling for servers, longer support timelines, and compatibility with other applications that may use double-precision math.

• The most important metric of a GPU is the device VRAM memory bandwidth. This is the rate at which the GPU can read and write from its own memory. This is generally more important than GPU core count or clock speed, as almost all operations on the GPU are memory-bandwidth limited. When selecting GPUs, this is the primary metric to compare (along with price). For example, the NVIDIA RTX 3090 has 24GB of memory at 936 GB/s bandwidth, the NVIDIA A100 has up to 80GB memory at 1935 GB/s bandwidth, and the NVIDIA A4000 has 16GB at 448 GB/s.

• GPU memory size is the main limiting factor in terms of the box-sizes that can be handled during a 3D refinement. Other than this, memory size does not have any impact on speed. 11GB consumer cards can generally handle all processing steps (including motion correction of K3 data, etc) for particle box sizes up to 600^3.

• GPU-CPU interconnect bandwidth (eg. PCIE) is generally not a bottleneck (e.g., for most job types, we get similar benchmark performance on 8x or 16x PCIE lanes) but IO bandwidth reading data from cluster storage/local SSD is usually a significant factor in performance. This is especially true for preprocessing and CryoSPARC Live, as movies, micrographs, and particles need to be read, written, and transferred rapidly to keep up with collection and GPUs can process the data very quickly.

• In many cases, older or slower GPUs can often perform almost equally as well as the newest, fastest GPUs because most computations in CryoSPARC are not bottlenecked by GPU compute speed, but rather by GPU memory bandwidth and disk I/O speed.

Browser Requirements

The CryoSPARC web interface works best on the latest version of Google Chrome. Firefox and Safari are also an option, although some features may not work as intended. Internet Explorer is not supported. See this guide for more information on accessing the CryoSPARC web interface.

Additional Configuration Notes

Network Accessibility

Network security must be an important factor in the installation and ongoing management of any CryoSPARC instance. Access to the network that hosts a CryoSPARC instance must be carefully controlled, as CryoSPARC instances are not security-hardened against malicious actors on the network.

CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances should never be directly hosted on the internet or an untrusted network, without a separate controlled authentication layer.

CryoSPARC’s User Interface does include a user management system, and CryoSPARC user accounts and passwords help control access to the interface within a trusted private network, but please note that CryoSPARC passwords are not intended as a barrier against malicious access.

Root Access

The CryoSPARC system is specifically designed not to require root access to install or use. The reason for this is to avoid security vulnerabilities that can occur when a network application (web interface, database, etc.,) is hosted as the root user. For this reason, the CryoSPARC system must be installed and run as a regular UNIX user (cryosparcuser), and all input and output file locations must be readable and writable as this user. In particular, this means that project input and output directories that are stored within a regular user's home directory need to be accessible by cryosparcuser, or else (more commonly) another location on a shared file system must be used for CryoSPARC project directories.

Multi-user environment

If you are installing the CryoSPARC system for use by many users (for example within a lab), there are two options:

Using UNIX Groups

Create a new regular user (cryosparcuser) and install and run CryoSPARC as this user. Create a CryoSPARC project directory (on a shared file system) where project data will be stored, and create sub-directories for each lab member. If extra security is necessary, use UNIX group privileges to make each sub-directory read/writeable only by cryosparcuser and the appropriate lab member's UNIX account. Within the CryoSPARC command-line interface, create a CryoSPARC user account for each lab member, and have each lab member create their projects within their respective project directories. This method relies on the CryoSPARC web application for security to limit each user to see only their own projects. This is not guaranteed security, and malicious users who try hard enough will be able to modify the system to be able to see the projects and results of other users.

Using Separate CryoSPARC Instances

If each user must be guaranteed complete isolation and security of their projects, each user must install CryoSPARC independently within their own home directories. Projects can be kept private within user home directories as well, using UNIX permissions. Multiple single-user CryoSPARC master processes can be run on the same master node, and they can all submit jobs to the same cluster scheduler system. This method relies on the UNIX system for security and is more tedious to manage but provides stronger access restrictions. Each user will need to have their own CryoSPARC license ID in this case.

Deploying CryoSPARC on AWS

CryoSPARC can be deployed on-premises or in the cloud. See below for a guide on deploying CryoSPARC on AWS resources.

Database and Command API Security

CryoSPARC v4.0 introduces additional authentication to reduce the likelihood of accidental mis-use by actors on a large institution/multi-user shared network.

Database Security

CryoSPARC's MongoDB database runs with access control enabled: Requests to read or write from the database must be authenticated with a username and password. CryoSPARC sets this password automatically and uses it for internal master → master and worker → master requests. Note that communication between the database and other CryoSPARC services is not encrypted.

Enable or disable MongoDB access control by setting CRYOSPARC_DB_ENABLE_AUTH variable in cryosparc_master/config.sh and cryosparc_worker/config.sh to true(default) or false.

To connect to the database with access control, use cryosparcm mongo to access the Mongo shell, or use cryosparcm icli to access an interactive Python client.

Command API Security

CryoSPARC's command API server (executes actions triggered by the web application including creating and modifying projects and jobs) also requires authentication.

The command server expects the CryoSPARC License ID in theLicense-ID header of incoming web requests. CryoSPARC includes this automatically in internal requests to the API. Requests with missing or incorrect license ID will be rejected. Note that communication between the command server and other CryoSPARC services is not encrypted.

You provide the License ID during CryoSPARC master and worker package installation. The license is written in plain-text to config.sh in the installation directories. The license ID in the worker installation must match the master installation.

Optional password argument

CryoSPARC allows creating users and updating users from the command line. Rather than specifying a --password flag during these operations, you may omit it to instead run a secure password prompt.

Example Systems

We do not currently partner with any specific hardware vendors to sell machines with CryoSPARC pre-installed.

Example Hardware Systems

Below are details of example workstations that meet or exceed the minimum requirements specified above, including those we use internally for development and testing.