CryoSPARC Architecture and System Requirements
Description of CryoSPARC HPC software system architecture, typical setups (e.g., workstation, cluster).
Last updated
Description of CryoSPARC HPC software system architecture, typical setups (e.g., workstation, cluster).
Last updated
CryoSPARC is a backend and frontend high-performance computing software system that provides data processing and image analysis capabilities for single particle cryo-EM, along with a rich browser-based user interface and command line tools.
CryoSPARC can be deployed on-premises or in the cloud.
CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances are not security-hardened against malicious actors on the network and should never be hosted directly on the internet or an untrusted network without a separate controlled authentication layer.
The system is based on a master-worker pattern.
The master processes (web application, core application and MongoDB database) run together on one machine (master node). The master node requires relatively lightweight resources (4+ CPUs, 16GB+ RAM, 250GB+ HDD storage)
Worker processes run on any available/configured machine that has NVIDIA GPUs (worker node). The worker is responsible for all actual computation and data handling and is dispatched by the master node.
The same node can function as both master and worker.
The master-worker architecture allows CryoSPARC to be installed and scaled up flexibly on a variety of hardware, including a single workstation, groups of workstations, cluster nodes, HPC clusters, cloud nodes, and more.
CryoSPARC can support a heterogeneous mixture of all typical setups in a single instance. This means you can start with installing CryoSPARC on a single workstation, then connect a worker node or cluster as your data processing requirements scale.
Both the CryoSPARC master and CryoSPARC worker processes may run on the same machine. The only requirement is that GPU resources are available for the CryoSPARC worker processes. This is the simplest setup.
In the master-worker setup, the CryoSPARC master is installed on a lightweight machine, and the worker processes are installed on one or more GPU servers. This is the most flexible setup for installing CryoSPARC. There are three main requirements for this setup, which are also explained in greater detail in the installation sections of this document:
1) All nodes have access to a shared file system. This file system is where the project directories are located, allowing all nodes to read and write intermediate results as jobs start and complete.
2) The master node has password-less SSH access to each of the worker nodes. SSH is used to execute jobs on the worker nodes from the master node.
3) All worker nodes have TCP access to 10 consecutive ports on the master node (default ports are 39000-39009). These ports are used for metadata communication via HTTP Remote Procedure Call (RPC) based API requests.
It is also possible to use one of the worker nodes as the master, in which case no standalone master node is necessary. Under high loads, this can lead to instability if the GPU worker node hangs or runs out of RAM, causing the master processes running the web application and database to also hang.
The master node can also spawn or submit jobs to a cluster scheduler system (e.g., Slurm Workload Manager). This integration is transparent, and works similar to the master-worker setup explained above, except all resource scheduling is handled by the cluster scheduler, and CryoSPARC's scheduler is only used for orchestration and management of jobs. Similar requirements are present:
1) All nodes have access to a shared file system. This file system is where the project directories are located, allowing all nodes to read and write results as jobs start and complete.
2) All worker nodes have TCP access to 10 consecutive ports on the master node (default ports are 39000-39009). These ports are used for metadata communication via HTTP Remote Procedure Call (RPC) based API requests.
For a cluster setup, the master node can be a regular cluster node (or even a login node) if this makes networking requirements easier, but the CryoSPARC master processes must be run continuously. If the master is to be run on a regular cluster node, the node may need to be requested from your scheduler in interactive mode or for an indefinitely running job.
Project directories are created in locations specified by CryoSPARC users. If administering a multi-user cluster instance, ensure that users create project directories in locations where both the master and worker nodes have access.
CryoSPARC supports most cluster schedulers, including SLURM, SGE and PBS. Please see here for example cluster configurations for popular schedulers.
CryoSPARC Cluster Integration Script ExamplesThe following are requirements for every master and worker node in the system unless otherwise specified.
| Component | Requirement |
|---|---|
Architecture | x86-64 (Intel or AMD) |
Operating System | Modern Linux OS (Ubuntu is recommended). Please see the section on Operating System for more details and limitations. |
Shell | |
User Account |
|
Software | Nvidia driver (worker nodes only). See details on Nvidia and CUDA requirements. |
Filesystem | Shared file system across all nodes |
The following are requirements specific to the master node.
Component | Minimum Requirement | Recommended |
CPU | 4+ cores | 8+ cores at 2.8GHz+ |
RAM | 16GB+ | 32GB DDR4 |
System Storage | 250GB+ HDD | 500GB SSD |
Fast Local Storage | Not Required | Not Required |
GPU | Not Required | Not Required |
Network | 1Gbps link to storage servers | 10Gbps link to storage servers |
A 10Gbps connection is recommended to the storage servers given raw cryo-EM movies can be several TB in size, and I/O bottlenecks are more of a concern than processing power for pre-processing jobs in CryoSPARC.
Although a CPU with a higher core count is recommended, a CPU with a faster clock rate is more advantageous due to how master processes are implemented.
Enough System Storage is required to host the cryosparc_master installation package and database folder. Each CryoSPARC project occupies between 100MB and 5GB of database storage, depending on the size of the project. 500GB is enough for approximately 200 medium-sized projects. Note that this excludes the space required for CryoSPARC project data in bulk storage, which could be in terabytes for larger projects.
The following are requirements for each worker node/cluster worker.
Component | Minimum Requirement | Recommended |
CPU | 2+ cores per GPU | 4 cores per GPU |
CPU Memory Bandwidth | 50+ GB/s | 100+ GB/s |
RAM | 32GB+ per GPU | 64GB DDR4 per GPU |
System Storage | 25GB+ HDD | 50GB+ SSD |
Fast Local Storage | 1TB SSD | 2TB PCIe SSD |
GPU | 1+ NVIDIA GPU with CC 3.5+, 11GB+ VRAM | 1+ NVIDIA Tesla V100, RTX2080Ti, RTX3090, etc |
Network | 1Gbps link to storage servers | 10Gbps link to storage servers |
High CPU memory bandwidth is especially important for CryoSPARC Live preprocessing.
System RAM is very important for worker nodes and should scale proportionately to the number of GPUs available for processing on the system.
Enough System Storage is required to host the cryosparc_worker installation package.
Fast local storage is also necessary as reconstruction jobs require random access to particle images. SSDs provide high throughput in this context. See the section on Solid State Storage for more details.
Currently, Ubuntu Desktop 16+ is the best operating system to use with CryoSPARC, as extensive testing is carried out on this platform before every release. Ubuntu 22.04 is not supported in CryoSPARC v3.4.0 and earlier, but is supported in CryoSPARC v4.0.0 and later.
There are various unresolved incompatibilities and instabilities when using CUDA applications on older Linux kernels (e.g., 3.10 found in CentOS 7). Several users have reported sporadic occurrences of cufftInvalidPlan and cuMemHostAlloc Failed errors while using CentOS 7.
Fast disks are a necessity for processing cryo-EM data efficiently. Fast sequential read/write throughput is needed during pre-processing stages (e.g., motion correction) where the volume of data is very large (tens of TB) while the amount of computation is relatively low (sequential processing for motion correction, CTF estimation, particle picking, etc.)
Spinning disk arrays in a RAID configuration are used to store large raw data files, and often cluster file systems are used for larger systems. As a rule of thumb, to saturate a 4-GPU machine during pre-processing, a sustained sequential read of 1000MB/s is required.
Compression can greatly reduce the amount of data stored in movie files, and also greatly speeds up preprocessing because decompression is actually faster than reading uncompressed data straight from disk. Typically, counting-mode movie files are stored in LZW compressed TIFF format without gain correction, so that the gain reference file is stored separately and must be applied on-the-fly during processing (which is supported by CryoSPARC). Compressing gain corrected movies can often result in much worse compression ratios than compressing pre-gain corrected (integer count) data.
CryoSPARC supports LZW compressed TIFF format, EER format and BZ2 compressed MRC format natively. In either case, the gain reference must be supplied as an MRC file. TIFF, EER and BZ2 compression are implemented as multi-core decompression streams on-the-fly.
SSD space is optional on a per-worker node basis but is highly recommended for worker nodes that will be running refinements and reconstructions using particle images. Nodes reserved for pre-processing (motion correction, particle picking, CTF estimation, etc) do not need to have an SSD.
CryoSPARC particle processing algorithms rely on random-access patterns and multiple passes through the data, rather than sequentially reading the data at once. Using a storage medium that allows for fast random reads will speed up processing dramatically.
CryoSPARC manages the SSD cache on each worker node transparently. Files are automatically cached, re-used across the same project and deleted if more space is needed. Please see the SSD Caching guide for more information.
The size of your typical single particle cryo-EM datasets will inform the size of SSD you choose to use. For a sample calculation, see:
At least one worker node must have GPUs available to run the complete set of CryoSPARC jobs. Non-GPU workers may run CPU-only jobs.
The GPU memory (VRAM) in each GPU limits the maximum particle box size for reconstruction. Typically, a GPU with 12GB VRAM can handle a box size of up to 700^3, and up to 1024^3 in some job types. See the following tutorial for more details.
Tutorial: Maximum Box Sizes for RefinementPlease ensure each connected worker includes a recent version of the Nvidia Driver compatible with your GPU. See this section for details. Download the latest driver for your GPUs here. Visit Troubleshooting to resolve common GPU errors.
When acquiring GPUs to use with CryoSPARC, the following considerations may be useful.
CryoSPARC almost exclusively uses single-precision operations on the GPU. As such, consumer cards generally have a much better price/performance ratio than enterprise cards. Enterprise cards do have their own benefits such as reliability, better cooling for servers, longer support timelines, and compatibility with other applications that may use double-precision math.
The most important metric of a GPU is the device VRAM memory bandwidth. This is the rate at which the GPU can read and write from its own memory. This is generally more important than GPU core count or clock speed, as almost all operations on the GPU are memory-bandwidth limited. When selecting GPUs, this is the primary metric to compare (along with price). For example, the NVIDIA RTX 3090 has 24GB of memory at 936 GB/s bandwidth, the NVIDIA A100 has up to 80GB memory at 1935 GB/s bandwidth, and the NVIDIA A4000 has 16GB at 448 GB/s.
GPU memory size is the main limiting factor in terms of the box-sizes that can be handled during a 3D refinement. Other than this, memory size does not have any impact on speed. 11GB consumer cards can generally handle all processing steps (including motion correction of K3 data, etc) for particle box sizes up to 600^3.
GPU-CPU interconnect bandwidth (eg. PCIE) is generally not a bottleneck (e.g., for most job types, we get similar benchmark performance on 8x or 16x PCIE lanes) but IO bandwidth reading data from cluster storage/local SSD is usually a significant factor in performance. This is especially true for preprocessing and CryoSPARC Live, as movies, micrographs, and particles need to be read, written, and transferred rapidly to keep up with collection and GPUs can process the data very quickly.
In many cases, older or slower GPUs can often perform almost equally as well as the newest, fastest GPUs because most computations in CryoSPARC are not bottlenecked by GPU compute speed, but rather by GPU memory bandwidth and disk I/O speed.
The CryoSPARC web interface works best on the latest version of Google Chrome. Firefox and Safari are also an option, although some features may not work as intended. Internet Explorer is not supported. See this guide for more information on accessing the CryoSPARC web interface.
Network security must be an important factor in the installation and ongoing management of any CryoSPARC instance. Access to the network that hosts a CryoSPARC instance must be carefully controlled, as CryoSPARC instances are not security-hardened against malicious actors on the network.
CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances should never be directly hosted on the internet or an untrusted network, without a separate controlled authentication layer.
CryoSPARC’s User Interface does include a user management system, and CryoSPARC user accounts and passwords help control access to the interface within a trusted private network, but please note that CryoSPARC passwords are not intended as a barrier against malicious access.
The CryoSPARC system is specifically designed not to require root access to install or use. The reason for this is to avoid security vulnerabilities that can occur when a network application (web interface, database, etc.,) is hosted as the root user. For this reason, the CryoSPARC system must be installed and run as a regular UNIX user (cryosparcuser), and all input and output file locations must be readable and writable as this user. In particular, this means that project input and output directories that are stored within a regular user's home directory need to be accessible by cryosparcuser, or else (more commonly) another location on a shared file system must be used for CryoSPARC project directories.
If you are installing the CryoSPARC system for use by many users (for example within a lab), there are two options:
Create a new regular user (cryosparcuser) and install and run CryoSPARC as this user. Create a CryoSPARC project directory (on a shared file system) where project data will be stored, and create sub-directories for each lab member. If extra security is necessary, use UNIX group privileges to make each sub-directory read/writeable only by cryosparcuser and the appropriate lab member's UNIX account. Within the CryoSPARC command-line interface, create a CryoSPARC user account for each lab member, and have each lab member create their projects within their respective project directories. This method relies on the CryoSPARC web application for security to limit each user to see only their own projects. This is not guaranteed security, and malicious users who try hard enough will be able to modify the system to be able to see the projects and results of other users.
If each user must be guaranteed complete isolation and security of their projects, each user must install CryoSPARC independently within their own home directories. Projects can be kept private within user home directories as well, using UNIX permissions. Multiple single-user CryoSPARC master processes can be run on the same master node, and they can all submit jobs to the same cluster scheduler system. This method relies on the UNIX system for security and is more tedious to manage but provides stronger access restrictions. Each user will need to have their own CryoSPARC license ID in this case.
CryoSPARC can be deployed on-premises or in the cloud. See below for a guide on deploying CryoSPARC on AWS resources.
Deploying CryoSPARC on AWSThe information in this section, Database and Command API Security, applies to CryoSPARC v4.0+.
CryoSPARC v4.0 introduces additional layers of security for the CryoSPARC application to reduce the likelihood of accidental or intentional mis-use of the software system components by actors on a large institution/multi-user shared network. Note that despite these improvements in v4.0, CryoSPARC is still designed to be run only within a trusted private network. CryoSPARC instances are not security-hardened against malicious actors on the network and should never be hosted directly on the internet or an untrusted network without a separate controlled authentication layer.
Starting in v4.0, the MongoDB database maintained by CryoSPARC runs with access control enabled. This means that requests to read or write from the database must be authenticated with a username and password. CryoSPARC sets this password automatically and uses it for internal requests to the database. Note that requests to the database are not encrypted.
MongoDB access control for your CryoSPARC instance can be toggled by changing the CRYOSPARC_DB_ENABLE_AUTH variable in config.sh, which defaults to true when installing or upgrading to CryoSPARC v4+.
Access control for MongoDB can be disabled by changing the value of CRYOSPARC_DB_ENABLE_AUTH to false in config.sh.
To connect to the database with access control enabled, you can use cryosparcm mongo to access the Mongo shell, or use cryosparcm icli to access an interactive database client.
Starting in v4.0, the CryoSPARC command API (that executes actions triggered by the web application including creating and modifying projects and jobs) also requires authentication. Note that requests to the database are not encrypted.
The CryoSPARC command server expects the CryoSPARC License ID to be included in the headers of incoming requests. CryoSPARC includes this automatically in internal requests to the APIs. Requests without the license ID in the header or with a license ID different from the one used by the server will be rejected. The license ID is expected in the header key License-ID.
At the time of installation, the License ID is provided to both the master and worker installations and written into the corresponding [config.sh](<http://config.sh>) files in the master and worker installation directories. Since the CryoSPARC worker must make API calls, the license ID in the worker installation must match the master installation.
Starting in v4.0, password inputs through the command line for CryoSPARC features such as installing, creating users, and updating users can be done without using the --password flag. These features will now prompt the user to securely enter a password after the command is used.
We do not currently partner with any specific hardware vendors to sell machines with CryoSPARC pre-installed.
Below are details of example workstations that meet or exceed the minimum requirements specified above, including those we use internally for development and testing.
Component | Hardware Product |
CPU | 32 Cores (base clock 2.8GHz+), e.g, AMD Threadripper 3975X |
Memory | 256GB DDR4 @ 3200MHz |
Storage | 4TB PCIe SSD (cache); 200TB RAID 6 storage server via 10Gbps link (raw movies) |
GPU | 4x NVIDIA Quadro GV100, or 4x NVIDIA Tesla V100 or 4x NVIDIA RTX 8000 |
