CryoSPARC Architecture and System Requirements
Description of CryoSPARC HPC software system architecture, typical setups (e.g., workstation, cluster).
Last updated
Description of CryoSPARC HPC software system architecture, typical setups (e.g., workstation, cluster).
Last updated
CryoSPARC is a backend and frontend high-performance computing software system that provides data processing and image analysis capabilities for single particle cryo-EM, along with a rich browser-based user interface and command line tools.
CryoSPARC can be deployed on-premises or in the cloud.
CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances are not security-hardened against malicious actors on the network and should never be hosted directly on the internet or an untrusted network without a separate controlled authentication layer.
The system is based on a master-worker pattern.
The master processes (web application, core application and MongoDB database) run together on one machine (master node). The master node requires relatively lightweight resources (4+ CPUs, 16GB+ RAM, 250GB+ HDD storage)
Worker processes run on any available/configured machine that has NVIDIA GPUs (worker node). The worker is responsible for all actual computation and data handling and is dispatched by the master node.
The master-worker architecture allows CryoSPARC to be installed and scaled up flexibly on a variety of hardware, including a single workstation, groups of workstations, cluster nodes, HPC clusters, cloud nodes, and more.
Both the CryoSPARC master and CryoSPARC worker processes may run on the same machine. The only requirement is that GPU resources are available for the CryoSPARC worker processes. This is the simplest setup.
1) All nodes have access to a shared file system. This file system is where the project directories are located, allowing all nodes to read and write intermediate results as jobs start and complete.
2) The master node has password-less SSH access to each of the worker nodes. SSH is used to execute jobs on the worker nodes from the master node.
3) All worker nodes have TCP access to 10 consecutive ports on the master node (default ports are 39000-39009). These ports are used for metadata communication via HTTP Remote Procedure Call (RPC) based API requests.
1) All nodes have access to a shared file system. This file system is where the project directories are located, allowing all nodes to read and write results as jobs start and complete.
2) All worker nodes have TCP access to 10 consecutive ports on the master node (default ports are 39000-39009). These ports are used for metadata communication via HTTP Remote Procedure Call (RPC) based API requests.
CryoSPARC supports most cluster schedulers, including SLURM, SGE and PBS. Please see here for example cluster configurations for popular schedulers.
The following are requirements for every master and worker node in the system unless otherwise specified.
Architecture
x86-64 (Intel or AMD)
Operating System
Shell
User Account
cryosparcuser
Software
Filesystem
Shared file system across all nodes
The following are requirements specific to the master node.
Component
Minimum Requirement
Recommended
CPU
4+ cores
8+ cores at 2.8GHz+
RAM
16GB+
32GB DDR4
System Storage
250GB+ HDD
500GB SSD
Fast Local Storage
Not Required
Not Required
GPU
Not Required
Not Required
Network
1Gbps link to storage servers
10Gbps link to storage servers
A 10Gbps connection is recommended to the storage servers given raw cryo-EM movies can be several TB in size, and I/O bottlenecks are more of a concern than processing power for pre-processing jobs in CryoSPARC.
Although a CPU with a higher core count is recommended, a CPU with a faster clock rate is more advantageous due to how master processes are implemented.
Enough System Storage is required to host the cryosparc_master installation package and database folder. Each CryoSPARC project occupies between 100MB and 5GB of database storage, depending on the size of the project. 500GB is enough for approximately 200 medium-sized projects. Note that this excludes the space required for CryoSPARC project data in bulk storage, which could be in terabytes for larger projects.
The following are requirements for each worker node/cluster worker.
Component
Minimum Requirement
Recommended
CPU
2+ cores per GPU
4 cores per GPU
CPU Memory Bandwidth
50+ GB/s
100+ GB/s
RAM
32GB+ per GPU
64GB DDR4 per GPU
System Storage
25GB+ HDD
50GB+ SSD
Fast Local Storage
1TB SSD
2TB PCIe SSD
GPU
1+ NVIDIA Tesla V100, RTX2080Ti, RTX3090, etc
Network
1Gbps link to storage servers
10Gbps link to storage servers
System RAM is very important for worker nodes and should scale proportionately to the number of GPUs available for processing on the system.
Enough System Storage is required to host the cryosparc_worker installation package.
Fast disks are a necessity for processing cryo-EM data efficiently. Fast sequential read/write throughput is needed during pre-processing stages (e.g., motion correction) where the volume of data is very large (tens of TB) while the amount of computation is relatively low (sequential processing for motion correction, CTF estimation, particle picking, etc.)
Spinning disk arrays in a RAID configuration are used to store large raw data files, and often cluster file systems are used for larger systems. As a rule of thumb, to saturate a 4-GPU machine during pre-processing, a sustained sequential read of 1000MB/s is required.
Compression can greatly reduce the amount of data stored in movie files, and also greatly speeds up preprocessing because decompression is actually faster than reading uncompressed data straight from disk. Typically, counting-mode movie files are stored in LZW compressed TIFF format without gain correction, so that the gain reference file is stored separately and must be applied on-the-fly during processing (which is supported by CryoSPARC). Compressing gain corrected movies can often result in much worse compression ratios than compressing pre-gain corrected (integer count) data.
CryoSPARC supports LZW compressed TIFF format, EER format and BZ2 compressed MRC format natively. In either case, the gain reference must be supplied as an MRC file. TIFF, EER and BZ2 compression are implemented as multi-core decompression streams on-the-fly.
SSD space is optional on a per-worker node basis but is highly recommended for worker nodes that will be running refinements and reconstructions using particle images. Nodes reserved for pre-processing (motion correction, particle picking, CTF estimation, etc) do not need to have an SSD.
CryoSPARC particle processing algorithms rely on random-access patterns and multiple passes through the data, rather than sequentially reading the data at once. Using a storage medium that allows for fast random reads will speed up processing dramatically.
The size of your typical single particle cryo-EM datasets will inform the size of SSD you choose to use. For a sample calculation, see:
At least one worker node must have GPUs available to run the complete set of CryoSPARC jobs. Non-GPU workers may run CPU-only jobs.
The GPU memory (VRAM) in each GPU limits the maximum particle box size for reconstruction. Typically, a GPU with 12GB VRAM can handle a box size of up to 700^3, and up to 1024^3 in some job types. See the following tutorial for more details.
When acquiring GPUs to use with CryoSPARC, the following considerations may be useful.
CryoSPARC almost exclusively uses single-precision operations on the GPU. As such, consumer cards generally have a much better price/performance ratio than enterprise cards. Enterprise cards do have their own benefits such as reliability, better cooling for servers, longer support timelines, and compatibility with other applications that may use double-precision math.
The most important metric of a GPU is the device VRAM memory bandwidth. This is the rate at which the GPU can read and write from its own memory. This is generally more important than GPU core count or clock speed, as almost all operations on the GPU are memory-bandwidth limited. When selecting GPUs, this is the primary metric to compare (along with price). For example, the NVIDIA RTX 3090 has 24GB of memory at 936 GB/s bandwidth, the NVIDIA A100 has up to 80GB memory at 1935 GB/s bandwidth, and the NVIDIA A4000 has 16GB at 448 GB/s.
GPU memory size is the main limiting factor in terms of the box-sizes that can be handled during a 3D refinement. Other than this, memory size does not have any impact on speed. 11GB consumer cards can generally handle all processing steps (including motion correction of K3 data, etc) for particle box sizes up to 600^3.
GPU-CPU interconnect bandwidth (eg. PCIE) is generally not a bottleneck (e.g., for most job types, we get similar benchmark performance on 8x or 16x PCIE lanes) but IO bandwidth reading data from cluster storage/local SSD is usually a significant factor in performance. This is especially true for preprocessing and CryoSPARC Live, as movies, micrographs, and particles need to be read, written, and transferred rapidly to keep up with collection and GPUs can process the data very quickly.
In many cases, older or slower GPUs can often perform almost equally as well as the newest, fastest GPUs because most computations in CryoSPARC are not bottlenecked by GPU compute speed, but rather by GPU memory bandwidth and disk I/O speed.
Network security must be an important factor in the installation and ongoing management of any CryoSPARC instance. Access to the network that hosts a CryoSPARC instance must be carefully controlled, as CryoSPARC instances are not security-hardened against malicious actors on the network.
CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances should never be directly hosted on the internet or an untrusted network, without a separate controlled authentication layer.
CryoSPARC’s User Interface does include a user management system, and CryoSPARC user accounts and passwords help control access to the interface within a trusted private network, but please note that CryoSPARC passwords are not intended as a barrier against malicious access.
The CryoSPARC system is specifically designed not to require root access to install or use. The reason for this is to avoid security vulnerabilities that can occur when a network application (web interface, database, etc.,) is hosted as the root user. For this reason, the CryoSPARC system must be installed and run as a regular UNIX user (cryosparcuser), and all input and output file locations must be readable and writable as this user. In particular, this means that project input and output directories that are stored within a regular user's home directory need to be accessible by cryosparcuser, or else (more commonly) another location on a shared file system must be used for CryoSPARC project directories.
If you are installing the CryoSPARC system for use by many users (for example within a lab), there are two options:
Create a new regular user (cryosparcuser) and install and run CryoSPARC as this user. Create a CryoSPARC project directory (on a shared file system) where project data will be stored, and create sub-directories for each lab member. If extra security is necessary, use UNIX group privileges to make each sub-directory read/writeable only by cryosparcuser and the appropriate lab member's UNIX account. Within the CryoSPARC command-line interface, create a CryoSPARC user account for each lab member, and have each lab member create their projects within their respective project directories. This method relies on the CryoSPARC web application for security to limit each user to see only their own projects. This is not guaranteed security, and malicious users who try hard enough will be able to modify the system to be able to see the projects and results of other users.
If each user must be guaranteed complete isolation and security of their projects, each user must install CryoSPARC independently within their own home directories. Projects can be kept private within user home directories as well, using UNIX permissions. Multiple single-user CryoSPARC master processes can be run on the same master node, and they can all submit jobs to the same cluster scheduler system. This method relies on the UNIX system for security and is more tedious to manage but provides stronger access restrictions. Each user will need to have their own CryoSPARC license ID in this case.
CryoSPARC can be deployed on-premises or in the cloud. See below for a guide on deploying CryoSPARC on AWS resources.
CryoSPARC is designed to run within a trusted private network. CryoSPARC instances are not security-hardened against malicious actors on the network and should never be exposed to the Internet or hosted on an untrusted network.
The information in this section, Database and Command API Security, applies to CryoSPARC v4.0+.
CryoSPARC v4.0 introduces additional authentication to reduce the likelihood of accidental mis-use by actors on a large institution/multi-user shared network.
CryoSPARC's MongoDB database runs with access control enabled: Requests to read or write from the database must be authenticated with a username and password. CryoSPARC sets this password automatically and uses it for internal master → master and worker → master requests. Note that communication between the database and other CryoSPARC services is not encrypted.
Enable or disable MongoDB access control by setting CRYOSPARC_DB_ENABLE_AUTH variable in cryosparc_master/config.sh and cryosparc_worker/config.sh to true(default) or false.
To connect to the database with access control, use cryosparcm mongo to access the Mongo shell, or use cryosparcm icli to access an interactive Python client.
CryoSPARC's command server (executes actions triggered by the web application including creating and modifying projects and jobs) also requires authentication.
The command server expects the CryoSPARC License ID in theLicense-ID header of incoming web requests. CryoSPARC includes this automatically in internal requests to the API. Requests with missing or incorrect license ID will be rejected. Note that communication between the command server and other CryoSPARC services is not encrypted.
You provide the License ID during CryoSPARC master and worker package installation. The license is written in plain-text to config.sh in the installation directories. The license ID in the worker installation must match the master installation.
CryoSPARC allows creating users and updating users from the command line. Rather than specifying a --password flag during these operations, you may omit it to instead run a secure password prompt.
We do not currently partner with any specific hardware vendors to sell machines with CryoSPARC pre-installed.
Below are details of example workstations that meet or exceed the minimum requirements specified above, including those we use internally for development and testing.
Component
Hardware Product
CPU
32 Cores (base clock 2.8GHz+), e.g, AMD Threadripper 3975X
Memory
256GB DDR4 @ 3200MHz
Storage
4TB PCIe SSD (cache); 200TB RAID 6 storage server via 10Gbps link (raw movies)
GPU
4x NVIDIA Quadro GV100, or 4x NVIDIA Tesla V100 or 4x NVIDIA RTX 8000
In the master-worker setup, the CryoSPARC master is installed on a lightweight machine, and the worker processes are installed on one or more GPU servers. This is the most flexible setup for installing CryoSPARC. There are three main requirements for this setup, which are also explained in greater detail in the :
The master node can also spawn or submit jobs to a cluster scheduler system (e.g., ). This integration is transparent, and works similar to the master-worker setup explained above, except all resource scheduling is handled by the cluster scheduler, and CryoSPARC's scheduler is only used for orchestration and management of jobs. Similar requirements are present:
Modern Linux OS ( is recommended). Please see the section on for more details and limitations.
Nvidia driver (worker nodes only). .
1+ NVIDIA GPU with , 11GB+ VRAM
High CPU memory bandwidth is especially important for .
Fast local storage is also necessary as reconstruction jobs require random access to particle images. SSDs provide high throughput in this context. See the section on for more details.
Currently, is the best operating system to use with CryoSPARC, as extensive testing is carried out on this platform before every release. Ubuntu 22.04 is not supported in CryoSPARC v3.4.0 and earlier, but is supported in CryoSPARC v4.0.0 and later.
There are various unresolved incompatibilities and instabilities when using CUDA applications on older Linux kernels (e.g., 3.10 found in CentOS 7). Several users have reported sporadic occurrences of and errors while using CentOS 7.
CryoSPARC manages the SSD cache on each worker node transparently. Files are automatically cached, re-used across the same project and deleted if more space is needed.
Please ensure each connected worker includes a recent version of the Nvidia Driver compatible with your GPU. See for details. . Visit to resolve common GPU errors.
The CryoSPARC web interface works best on the latest version of . and are also an option, although some features may not work as intended. Internet Explorer is not supported. for more information on accessing the CryoSPARC web interface.
