2. RESTRICTIONS. Licensee may not: (i) modify, enhance, reverse-engineer, decompile, disassemble or create derived forms of the Software; (ii) copy the Software; (iii) sell, sub-license, lease, assign, transmit, distribute or otherwise transfer rights in/to the Software; (iv) allow third-party use of Licensee's installation of the Software; or (v) pledge, hypothecate, alienate or otherwise encumber the Software to any third party. Use of the Software is restricted to the Intended Purpose only.
3. NO WARRANTY. The Software is provided "as is" without warranty of any kind. Licensor makes no representations, warranties or covenants to Licensee, either express or implied, with respect to the Software or with respect to any Confidential Information (as defined herein) disclosed to Licensee. Licensor specifically disclaims any implied warranty or condition of non-infringement, merchantable quality or fitness for a particular purpose. Licensee acknowledges that the Software is of an experimental nature, that no particular results can be guaranteed, and that it has been advised by Licensor to undertake its own due diligence with respect to all matters arising from the Agreement.
4. LIMITATION OF LIABILITY. In no event is Licensor liable for any damages on any basis, in contract, tort or otherwise, of any kind and nature whatsoever, arising in respect of this Agreement, howsoever caused, including damages of any kind and nature caused by Licensor’s negligence or by a fundamental breach of contract or any other breach of duty whatsoever. Licensee is advised to safeguard important data, use caution and not rely in any way on the correct functioning or performance of the Software and/or accompanying materials.
5. NO IMPROVEMENTS. Licensor is under no obligation to provide Improvements to the Software. "Improvements" means any improvements, updates, variations, modifications, alterations, additions, error corrections, enhancements, functional changes or other changes to the Software, including, without limitation: (i) improvements or upgrades to improve software efficiency and maintainability; (ii) improvements or upgrades to improve operational integrity and efficiency; (iii) changes or modifications to correct errors; and (iv) additional licensed computer programs to otherwise update the Software.
6. NO FUTURE ENTITLEMENT. Nothing in this Agreement shall be construed as creating any obligation on Licensor to continue to develop, commercialize, offer, make available or support (i) the Software; or (ii) any feature, functionality or Improvement as may be encompassed in the Software from time to time.
7. SUSPENSION. Licensor reserves the right to modify, suspend or discontinue, temporarily or permanently, the Software, with or without notice and without liability to Licensee.
8. OWNERSHIP. Licensor retains title to and ownership of the Software and any Improvements. Nothing in this Agreement shall be construed as granting any express or implied ownership rights to Licensee in respect of the Software, associated documentation or Confidential Information, including but not limited to any patent, copyright, trademark or other intellectual property right.
9. INTELLECTUAL PROPERTY. All Intellectual Property, Intellectual Property Rights and distribution rights associated with or arising from the Software or Licensor’s Confidential Information remain exclusively with Licensor. “Intellectual Property” includes, without limitation, all technical data, designs, specifications, software, data, drawings, plans, reports, patterns, models, prototypes, demonstration units, practices, inventions, methods and related technology, processes or other information, and all rights therein, including, without limitation, patents, copyrights, industrial designs, trade-marks and any registrations or applications for the same and all other rights of intellectual property therein, including any rights for which arise from the above items being treated by the Parties as trade secrets or confidential information (the rights being “Intellectual Property Rights”).
10. CONFIDENTIAL INFORMATION. “Confidential Information” means any and all confidential or proprietary information of Licensor or Licensee which may be exchanged between the Parties at any time prior to and during the term of this Agreement, including, without limitation, business and marketing information, technology, know-how, ideas, reports, techniques, methods, processes, uses, composites, skills, and configurations of any kind. Without limiting the generality of the foregoing, Licensor’s Confidential Information includes: (i) the Software, including its features, functionality, performance, application and use; (ii) the computer code underlying the Software, including source and compiled code and all associated documentation and files; (iii) information relating to the performance or quality of the Software; (iv) the details of any technical assistance provided to Licensee during the term of this Agreement; (v) any other products or service made available to Licensee by Licensor during the term of this Agreement; and (vi) information regarding Licensor’s business operations or research and development activities.\ Neither party shall: (i) disclose, either directly or indirectly, any Confidential Information or any part thereof belonging to the other party, to any person except as is specifically contemplated in this Agreement; or (ii) use any Confidential Information or any part thereof belonging to the other party, for any purpose except as is specifically contemplated in this Agreement. The obligations of confidentiality set forth herein shall not apply to the extent that the information: (i) was already known to the relevant party without restriction at the time the information was disclosed to such party; (ii) was generally available to the public or otherwise was part of the public domain at the time of its disclosure to the relevant party; (iii) became generally available to the public or otherwise part of the public domain after its disclosure to the relevant party through no act or omission of such party; or (iv) was disclosed to the relevant party without restriction by a third party who, to the best of such party's knowledge and belief, had no obligation not to disclose such information.
11. FEEDBACK. Licensee may communicate to Licensor, whether or not at Licensor’s request, suggestions and comments regarding the Software, including without limitation, performance, user interface, experiment results, and errors (collectively, “Feedback”). Licensor shall have worldwide, non-exclusive, perpetual, irrevocable, royalty-free, fully-paid up rights to use such Feedback. Without limiting the generality of the foregoing, Licensor shall have the unencumbered right to make, use, copy, modify, sell, distribute, sub-license, and create derivative works of/incorporating the Feedback as part of any product, technology, service, specification or other documentation and to publicly perform or display, import, broadcast, transmit, distribute, license, offer to sell, and sell, rent, lease or lend anonymized copies of the Feedback (and derivative works thereof) as part of any product.
12. PERFORMANCE DATA AND ANALYTICS. Licensor may collect usage and performance data relating to Licensee’s installation of the Software. including, without limitation, data relating to: (i) software use, including the number of users, projects and experiments associated with an installation; (ii) error information, including error messages and user-submitted feedback; (iii) performance data, including experiment run times and failed experiments; (iv) hardware utilization, including the number of active nodes and memory usage; and (v) license status information, including confirmation of valid license status.
13. TERMINATION. Licensor reserves the right to terminate this Agreement immediately and without notice in the event Licensee fails to comply with any provision of this Agreement. On termination of this Agreement, whether by reason of expiry or otherwise, Licensee shall promptly discontinue use of the Software, destroy its installation of the Software and, at Licensor's request, return the Software to Licensor at no cost to Licensor. Licensor may exercise any or or more of the remedies available to it under the terms of this Agreement, in addition to any remedy available at law. Failure of Licensor to enforce a right under this Agreement shall not act as a waiver of that right.
14. GOVERNING LAW. This Agreement is made in Ontario and governed by and construed in accordance with the laws of the Province of Ontario and the federal laws of Canada applicable therein. The Parties attorn to the exclusive jurisdiction of the Courts of the Province of Ontario.
15. SURVIVAL. The provisions of subsections 2-12 shall survive termination of this Agreement.
16. SEVERABILITY. If any term, covenant, condition or provision of this Agreement is held by a court of competent jurisdiction to be invalid, void or unenforceable, it is the Parties' intent that such provision be reduced in scope by the court only to the extent deemed necessary by that court to render the provision reasonable and enforceable and the remainder of the provisions of this Agreement will in no way be affected, impaired or invalidated as a result.
17. NO AGENCY. No provision of this Agreement or action by the Parties will establish or be deemed to establish any partnership, joint venture, principal-agent or employer-employee relationship in any way, or for any purpose, between Licensor and Licensee.
18. ENTIRE AGREEMENT. This Agreement including all schedules hereto, constitutes the entire agreement between the Parties concerning the subject matter hereof and supersedes all prior or collateral agreements, communications, representations, understandings, negotiations and discussions, oral or written. © 2020 Structura Biotechnology Inc. # CryoSPARC Architecture and System Requirements Description of CryoSPARC HPC software system architecture, typical setups (e.g., workstation, cluster). ## CryoSPARC System Architecture Overview CryoSPARC is a backend and frontend high-performance computing software system that provides data processing and image analysis capabilities for single particle cryo-EM, along with a rich browser-based user interface and command line tools. CryoSPARC can be deployed on-premises or in the cloud. {% hint style="warning" %} CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances are not security-hardened against malicious actors on the network and should never be hosted directly on the internet or an untrusted network without a separate controlled authentication layer. {% endhint %} ### Master-worker pattern The system is based on a master-worker pattern. * The master processes (web application, core application and MongoDB database) run together on one machine (**master** node). The master node requires relatively lightweight resources (4+ CPUs, 16GB+ RAM, 250GB+ HDD storage) * Worker processes run on any available/configured machine that has NVIDIA GPUs (**worker** node). The worker is responsible for all actual computation and data handling and is dispatched by the master node. {% hint style="info" %} The same node can function as both master and worker. {% endhint %} The master-worker architecture allows CryoSPARC to be installed and scaled up flexibly on a variety of hardware, including a single workstation, groups of workstations, cluster nodes, HPC clusters, cloud nodes, and more.  ## Typical CryoSPARC System Setups {% hint style="info" %} CryoSPARC can support a heterogeneous mixture of all typical setups in a single instance. This means you can start with installing CryoSPARC on a single workstation, then connect a worker node or cluster as your data processing requirements scale. {% endhint %} ### Single Workstation  Both the CryoSPARC **master** and CryoSPARC **worker** processes may run on the same machine. The only requirement is that GPU resources are available for the CryoSPARC worker processes. This is the simplest setup. ### Master-Worker  In the master-worker setup, the worker processes run on one or more GPU servers ("nodes"). The master processes may be run on a light-weight dedicated server or on a GPU server that is configured similarly to a single workstation (see [above](#single-workstation)). This is the most flexible setup for installing CryoSPARC. There are three main requirements for this setup, which are also explained in greater detail in the [installation sections of this document](/setup-configuration-and-management/cryosparc-installation-prerequisites): {% hint style="danger" %} **1) All nodes have access to a shared file system.** This file system is where the project directories are located, allowing all nodes to read and write intermediate results as jobs start and complete. **2) The master node** has password-less SSH access to each of the worker nodes. SSH is used to execute jobs on the worker nodes from the master node. **3) All worker nodes** have TCP access to 10 consecutive ports on the master node. These ports are used for metadata communication via HTTP API requests. {% endhint %} {% hint style="info" %} If master processes are run on a GPU server, heavy GPU processing loads can lead to instability if the GPU worker node hangs or runs out of RAM, causing the master processes running the web application and database to also hang. {% endhint %} In this *master-worker* pattern * processing jobs are queued to a specific scheduler *lane* * a scheduler *lane* is a collection of one or more (GPU) worker nodes * each worker node is associated with no more than one lane One may combine multiple worker nodes into a lane if there is no concern over which specific node will process a given CryoSPARC job. One may prefer single-node lanes if more direct control is desired over where a given job will run. This may be the case when hardware specifications, such as the amount of RAM or GPU models, vary significantly between nodes. A potential drawback is that a job queued to such a lane would remain queued to the selected lane even if resources are idle on another lane. ### Clusters  The master node can also spawn or submit jobs to a cluster scheduler system (e.g., [Slurm Workload Manager](https://slurm.schedmd.com/overview.html)). This integration is transparent, and works similar to the master-worker setup explained above, except all resource scheduling is handled by the cluster scheduler, and CryoSPARC's scheduler is only used for orchestration and management of jobs. Similar requirements are present: {% hint style="danger" %} **1) All nodes** have access to a shared file system. This file system is where the project directories are located, allowing all nodes to read and write results as jobs start and complete. **2) All worker nodes** have TCP access to 10 consecutive ports on the master node (default ports are 39000-39009). These ports are used for metadata communication via HTTP Remote Procedure Call (RPC) based API requests. {% endhint %} {% hint style="info" %} For a **cluster** setup, the master node can be a regular cluster node (or even a login node) if this makes networking requirements easier, but the CryoSPARC master processes must be run continuously. If the master is to be run on a regular cluster node, the node may need to be requested from your scheduler in interactive mode or for an indefinitely running job. {% endhint %} {% hint style="info" %} Project directories are created in locations specified by CryoSPARC users. If administering a multi-user cluster instance, ensure that users create project directories in locations where both the master and worker nodes have access. {% endhint %} #### Supported cluster schedulers CryoSPARC supports most cluster schedulers, including SLURM, SGE and PBS. Please see [here for more details about how CryoSPARC connects with a cluster system](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#connect-a-cluster-to-cryosparc). ## CryoSPARC System Requirements The following are requirements for every master and worker node in the system unless otherwise specified. | Component | Requirement | | ---------------: | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Architecture | x86-64 (Intel or AMD) | | Operating System | Modern Linux OS. Please see the section on [Operating System](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#operating-system) for more details and limitations. | | Shell | [Bash](https://www.gnu.org/software/bash/) | | User Account | `cryosparcuser` | | Software | Nvidia driver (worker nodes only). [See details on Nvidia and CUDA requirements](https://guide.cryosparc.com/setup-configuration-and-management/pages/-M7DHIJrIWYpsjbmcVFX#1.-nvidia-driver). | | Filesystem | Shared file system across all nodes | ### CryoSPARC Master Node Requirements The following are requirements specific to the master node. | Component | Minimum Requirement | Recommended | | ---------------------: | ----------------------------- | ------------------------------ | | **CPU** | 4+ cores | 8+ cores at 2.8GHz+ | | **RAM** | 16GB+ | 32GB DDR4 | | **System Storage** | 250GB+ HDD | 500GB SSD | | **Fast Local Storage** | Not Required | Not Required | | **GPU** | Not Required | Not Required | | **Network** | 1Gbps link to storage servers | 10Gbps link to storage servers | A 10Gbps connection is recommended to the storage servers given raw cryo-EM movies can be several TB in size, and I/O bottlenecks are more of a concern than processing power for pre-processing jobs in CryoSPARC. Although a CPU with a higher core count is recommended, a CPU with a faster clock rate is more advantageous due to how master processes are implemented. Enough System Storage is required to host the `cryosparc_master` installation package and database folder. Each CryoSPARC project occupies between 100MB and 5GB of database storage, depending on the size of the project. 500GB is enough for approximately 200 medium-sized projects. *Note that this excludes the space required for CryoSPARC project data in bulk storage, which could be in terabytes for larger projects.* ### Worker Node/Cluster Worker Minimum Requirements The following are requirements for each worker node/cluster worker. | Component | Minimum Requirement | Recommended | | -----------------------: | ---------------------------------------------------------------------------------------- | --------------------------------------------- | | **CPU** | 2+ cores **per GPU** | 4 cores **per GPU** | | **CPU Memory Bandwidth** | 50+ GB/s | 100+ GB/s | | **RAM** | 32GB+ **per GPU** | 64GB DDR4 **per GPU** | | **System Storage** | 25GB+ HDD | 50GB+ SSD | | **Fast Local Storage** | 1TB SSD | 2TB PCIe SSD | | **GPU** | 1+ NVIDIA GPU with [CC 3.5+](https://developer.nvidia.com/cuda-gpus#compute), 11GB+ VRAM | 1+ NVIDIA Tesla V100, RTX2080Ti, RTX3090, etc | | **Network** | 1Gbps link to storage servers | 10Gbps link to storage servers | High CPU memory bandwidth is especially important for [CryoSPARC Live preprocessing](/live/prerequisites-and-compute-resources-setup#preprocessing-lane). System RAM is very important for worker nodes and should scale proportionately to the number of GPUs available for processing on the system. Enough System Storage is required to host the `cryosparc_worker` installation package. Fast local storage is also necessary as reconstruction jobs require random access to particle images. SSDs provide high throughput in this context. See the section on [Solid State Storage](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#solid-state-storage-ssds) for more details. ### Operating System For CryoSPARC v5.0+, the operating system must support GLIBC 2.28 or greater. Therefore, the oldest compatible operating systems are Rocky/RHEL 8 and Ubuntu 20.04. For Ubuntu, version 22.04 or 24.04 is recommended. We recommend running CryoSPARC on an up-to-date long-term support Linux distribution, such as Ubuntu (22.04, 24.04), Rocky Linux (8, 9, 10) or a related distribution. {% hint style="warning" %} As of June 2026, CryoSPARC versions up to 5.0.6 are incompatible with the version 7 kernel that is the default for Ubuntu 26.04. {% endhint %} ### Disks and compression Fast disks are a necessity for processing cryo-EM data efficiently. Fast **sequential** read/write throughput is needed during **pre-processing** stages (e.g., motion correction) where the volume of data is very large (tens of TB) while the amount of computation is relatively low (sequential processing for motion correction, CTF estimation, particle picking, etc.) Spinning disk arrays in a RAID configuration are used to store large raw data files, and often cluster file systems are used for larger systems. As a rule of thumb, to saturate a 4-GPU machine during pre-processing, a **sustained sequential read of 1000MB/s is required**. Compression can greatly reduce the amount of data stored in movie files, and also greatly speeds up preprocessing because decompression is actually faster than reading uncompressed data straight from disk. Typically, counting-mode movie files are stored in LZW compressed TIFF format without gain correction, so that the gain reference file is stored separately and must be applied on-the-fly during processing (which is supported by CryoSPARC). Compressing gain corrected movies can often result in much **worse** compression ratios than compressing pre-gain corrected (integer count) data. CryoSPARC supports LZW compressed TIFF format, EER format and BZ2 compressed MRC format natively. In either case, the gain reference must be supplied as an MRC file. TIFF, EER and BZ2 compression are implemented as multi-core decompression streams on-the-fly. ### Solid State Storage (SSDs) SSD space is optional on a per-worker node basis but is **highly recommended** for worker nodes that will be running refinements and reconstructions using particle images. Nodes reserved for pre-processing (motion correction, particle picking, CTF estimation, etc) **do not** need to have an SSD. CryoSPARC particle processing algorithms rely on random-access patterns and multiple passes through the data, rather than sequentially reading the data at once. Using a storage medium that allows for fast random reads will speed up processing dramatically. CryoSPARC manages the SSD cache on each worker node transparently. Files are automatically cached, re-used across the same project and deleted if more space is needed. [Please see the SSD Caching guide for more information.](/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc) The size of your typical single particle cryo-EM datasets will inform the size of SSD you choose to use. For a sample calculation, see: {% embed url="
--worker <worker_hostname> = worker.cryosparc.hostname.com
# [required] hostname of the worker node to connect to your
# cryoSPARC instance
--master <master_hostname> = your.cryosparc.hostname.com
# [required] hostname of the server where the master is
# installed on
--port <port_number> = 61000
# [required] base port number for the cryoSPARC instance you
# are connecting the worker node to
--ssdpath <ssd_path>
# [optional] path to directory on local SSD
# replace this with --nossd to connect a worker node without an SSD
--update
# [optional] use when updating and existing configuration
--sshstr <custom_ssh_string>
# [optional] custom SSH connection string such as user@hostname
--cpus 4
# [optional] enable this number of CPU cores
--nogpu
# [optional] connect worker with no GPUs
--gpus 0,1,2,3
# [optional] enable specific GPU devices only
# For advanced configuration, run the gpulist command:
# $ bin/cryosparcw gpulist
# Detected 4 CUDA devices.
# id pci-bus name
# ---------------------------------------------------------------
# 0 0000:42:00.0 Quadro GV100
# 1 0000:43:00.0 Quadro GV100
# 2 0000:0B:00.0 Quadro RTX 5000
# 3 0000:0A:00.0 GeForce GTX 1080 Ti
# ---------------------------------------------------------------
# This will list the available GPUs on the worker node, and their
# corresponding numbers. Use this list to decide which GPUs you wish
# to enable using the --gpus flag, or leave this flag out to enable all GPUs.
--nossd
# [optional] connect a worker node with no SSD
--ssdquota <ssd_quota_mb>
# [optional] quota of how much SSD space to use (MB)
--ssdreserve <ssd_reserve_mb>
# [optional] minimum free space to leave on SSD (MB)
--lane <lane_name>
# [optional] name of lane to add worker to
--newlane
# [optional] force creation of a new lane if the lane specified by --lane
# does not exist
2. Copy the RSA public key to the remote compute node for password-less login: ``` ssh-copy-id remote_username@remote_hostname ``` *Note: `remote_username` and `remote_hostname` are your username and the hostname that you use to SSH into your compute node. This step will ask for your password.* ### Appendix B: Using SSH Forwarding with compression to reduce data usage Supply `-C` to the port tunnelling command to request compression of all data. This can help when downloading maps from the CryoSPARC UI, as masks can be greatly compressed. From `man ssh`: ``` -C Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11, TCP and UNIX-domain connections). The compression algorithm is the same used by gzip(1), and the “level” can be controlled by the CompressionLevel option for protocol version 1. Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. ``` For example: ``` ssh -N -f -L localhost:62222:localhost:61000 remote_hostname -C ``` ### Appendix C: Using Hardware Accelerated OpenSSH Ciphers If your system supports Intel or AMD AES-NI, you can take advantage of hardware accelerated ciphers that dramatically improve the performance of your SSH connection. To find out if your system supports this, [follow this tutorial.](https://www.cyberciti.biz/faq/how-to-find-out-aes-ni-advanced-encryption-enabled-on-linux-system/) If your system has these features enabled, supply the argument `-o Ciphers=aes128-gcm@openssh.com` or `-o Ciphers=aes256-gcm@openssh.com` (depending on what your system supports, but AES 256 is preferred) to the port forwarding command. For example: ``` ssh -N -f -L localhost:62222:localhost:61000 remote_hostname -C -o Ciphers=aes256-gcm@openssh.com ``` ### Appendix D: Custom SSL Certificate Authority Bundle cryoSPARC requires internet access from the main process to verify your license and perform updates. At minimum, CryoSPARC should have access to our license server at `https://get.cryosparc.com/`. On some older systems, or if your system is behind a HTTP proxy, CryoSPARC may have trouble getting the required SSL certificates to validate this requires. If you have a Certificate Authority (CA) bundle on your system, you may specify its path for CryoSPARC to use and apply. Add the following line to `cryosparc_master/config.sh` (substitute `/path/to/cabundle` with the path to the CA bundle on your system): ```bash export REQUESTS_CA_BUNDLE="/path/to/cabundle" ``` # (Optional) Hosting CryoSPARC Through a Reverse Proxy As discussed in [Accessing the CryoSPARC User Interface](/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc), there are various ways in which users can access the CryoSPARC web interface such as through a VPN connection or SSH tunnel. If you would like to host the CryoSPARC interface in a secure manner at a predictable URL, this can be done through a reverse proxy server. Reverse proxy servers allow for more control over how a user accesses a web application interface over other methods. By controlling incoming network traffic it is able to host the application at a static URL (for example `https://cryosparc.institution.edu`) and ensure all correspondence is secured via HTTPS. The method in which you host CryoSPARC through a reverse proxy is similar to hosting any other web application. However, the following serve as our recommended minimum requirements: * All incoming traffic should be served through HTTPS (via a SSL certificate) * HTTPS traffic requires a valid SSL certificate provided by a certificate authority (CA) for the domain in which you are hosting the interface. * If the server listens for incoming HTTP traffic, forward all connections to a more secure protocol (HTTPS) * Ensure traffic is also mediated by an organization-level authentication barrier (for example single sign-on). CryoSPARC should *not* be served via the public internet without any additional authentication checks. There are many ways to generate a SSL certificate for your domain, however, this will most likely be specific to your institution or organization. If you're unsure of how to generate a SSL certificate for your private network, please consult with your system or network administrator for guidance. Each institution or private network can have a specific setup requiring custom rules and/or proxy configuration considerations. Generally the example configurations below should be compatible with common reverse proxy installations. Please consult with your system or network administrator for guidance regarding institution-specific protocols for reverse proxy hosting. The following section will provide example configuration files for common reverse proxy servers given CryoSPARC is running on base port `61000`. ## NGINX This [NGINX](https://nginx.org/en/) configuration takes advantage of [authenticated origin pulls](https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/explanation/) for an added layer of security between the reverse-proxy and a downstream proxy/load balancer. ``` server { listen 80; listen [::]:80; server_name private.domain.dev; return 302 https://$server_name$request_uri; } server { # SSL configuration listen 443 ssl http2; listen [::]:443 ssl http2; ssl on; ssl_certificate /etc/certs/domain.dev/origin-cert.pem; ssl_certificate_key /etc/certs/domain.dev/private-key.pem; ssl_client_certificate /etc/certs/domain.dev/origin-pull-ca.pem; ssl_verify_client on; server_name private.domain.dev; access_log /var/log/nginx/private.domain.dev.access.log; error_log /var/log/nginx/private.domain.dev.error.log; location / { proxy_pass http://127.0.0.1:61000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header X-Forwarded-For $remote_addr; proxy_request_buffering off; proxy_buffering off; client_max_body_size 0; } } ``` An alternative configuration from our [Discussion Forum](https://discuss.cryosparc.com/t/how-to-run-cryosparc-on-an-https-url-using-an-ssl-certificate/2614/7) that serves the application over HTTPS and redirects incoming HTTP requests. ``` server { listen 80; server_name
Set this variable to override the SSD cache path provided when you installed the worker. Useful if the SSD cache path is generated by your cluster as an environment variable when the job is scheduled
Important: The worker must be connected with a stub SSD path for this to take effect, e.g., "cache\_path": "/tmp" in cluster\_config.json
| Variable | Description | Default Value |
|---|---|---|
CRYOSPARC_API_PROCS | Number of API service processes (1–8). | 3 |
CRYOSPARC_AUTO_EXPORT_INSTANCE_CONFIG_DIR | The location which instance configurations will be automatically exported to. | $ROOT_DIR/run |
CRYOSPARC_AUTO_EXPORT_INSTANCE_CONFIG_ENABLE | Enables automatic instance configuration exports. | true |
CRYOSPARC_AUTO_EXPORT_INSTANCE_CONFIG_INTERVAL_HOURS | Interval for the automatic export of instance configurations. | 1 |
CRYOSPARC_CLI_SKIP_ACCESS_CHECK | Set to 1 or true to disable filesystem permission checks for cryosparcm command-line arguments; may be used with network file systems where listed UNIX access permissions do not match the true available permissions. | false |
CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL | The amount of time to wait (in seconds) in between status updates for cluster jobs. | 10 |
CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES | The maximum amount of retries for cluster job status updates. | 1000000 |
CRYOSPARC_DB_CONNECTION_TIMEOUT_MS | The amount of time to wait (in milliseconds) for the database to respond when starting it. | 20000 |
CRYOSPARC_DB_ENABLE_AUTH | Enables authentication for MongoDB database operations | true |
CRYOSPARC_DB_MIN_SPACE_GB | Minimum free disk space required for MongoDB data directory. | 5 |
CRYOSPARC_DISABLE_EXTERNAL_REQUESTS | Disables application requests to external HTTPS resources used for information modules in the homepage including EMPIAR, EMDB, Discuss, CryoSPARC tutorials, and the CryoSPARC changelog. | false |
CRYOSPARC_FORCE_HOSTNAME | In master/worker or cluster modes, cryosparcm commands always run on the machine where cryosparc_master was initially installed. Set this to true to allow running on any machine | false |
CRYOSPARC_FORCE_USER | cryosparcm commands must be run by the same UNIX user account that owns the cryosparc_master installation. Set this to true to allow running cryosparcm from any user account | false |
CRYOSPARC_HEARTBEAT_SECONDS | CryoSPARC jobs running on worker nodes regularly report their status to the master command server to indicate that they are still running and active. If a job fails to report for more than this number of seconds (e.g., due to stalling, a slow network or a silent error), CryoSPARC marks the job as failed. Increase for very busy/low-resource worker nodes or slow/unreliable connections between the master and worker nodes. Increasing may reduce heartbeat-related job failures. | 180 |
CRYOSPARC_IGNORE_HIDDEN_FILES | When checking that job directory is empty before running a job, set to true to ignore any filenames that being with .. Use this if your file system or OS automatically creates hidden files | false |
CRYOSPARC_IGNORE_PORT_CONFLICTS | Ignore port conflicts when binding service ports. | false |
CRYOSPARC_INSECURE | Disables HTTPS secure connections and enables HTTP insecure connections, for connections to CryoSPARC license servers. | false |
CRYOSPARC_IO_URING | Set to false to force-disable io_uring for fast disk read operations during jobs. May be required for some older operating systems or systems with an incorrect io_uring implementation. | true |
CRYOSPARC_LICENSE_SERVER_ADDR | Override CryoSPARC license server address. Use for systems that require access through a proxy. | https://get.cryosparc.com |
CRYOSPARC_LIVE_RETRY_ATTEMPTS | Number of attempts to retry a failed worker job in CryoSPARC Live. If a worker job fails more times than this value, the job will not be automatically restarted. | 3 |
CRYOSPARC_LIVE_RETRY_TIMEOUT_SECONDS | Length of time to wait before restarting a failed worker job in CryoSPARC Live | 30 |
CRYOSPARC_MONGO_CACHE_GB | How much RAM to allocate for MongoDB database query cache, in GB | 4 |
CRYOSPARC_MOTION_CORRECTION_CPUS_PER_GPU | How many CPU cores to allocate for each GPU used in Patch, Full-frame and Local Motion Correction jobs | 6 |
CRYOSPARC_MOTION_CORRECTION_RAM_MB_PER_GPU | How much RAM (in MB) to allocate for each GPU used in Patch, Full-frame and Local Motion Correction jobs | 15000 |
CRYOSPARC_PROJECT_DIR_PREFIX | The prefix to add to a project directory name on the filesystem when it is created. | CS- |
CRYOSPARC_RETAIN_QUEUE_POSITION_SECONDS | When clearing and re-queuing a job, this value is the window of time for which the job will retain its position in the CryoSPARC instance job queue. i.e. The job will still count as being scheduled from when it was first queued for CRYOSPARC_RETAIN_QUEUE_POSITION_SECONDS long since it was cleared. After the window passes without re-queueing, the job will lose its position in the queue, and the next time it is queued it will schedule as normal. | 0 |
CRYOSPARC_SLACK_WEBHOOK_URL | If set, CryoSPARC makes an HTTP request to this URL each time a job's status changes with some job metadata encoded in JSON | - |
CRYOSPARC_SSD_CACHE_LIFETIME_DAYS | Whenever a job requires SSD cache, it automatically checks for and removes files that haven't been accessed in more than the number of days specified by this variable. Note: Files may remain on the SSD for longer than this amount since they only get cleaned up when a job runs. Files may remain on the SSD for shorter than this amount if they are not in use by an active job and additional space is needed for caching of other particles. | 30 |
REQUESTS_CA_BUNDLE | May be required for connection to the CryoSPARC license verification server on systems with outdated Certificate-Authority files or that filter HTTPS requests through a proxy. Specify a path to a file or folder that contains the certificates. | - |
| Variable | Description | Default Value |
|---|---|---|
CRYOSPARC_CACHE_COPY_BUFFER_SIZE_MB | Size of the buffer used when copying files to the CryoSPARC cache location. | 8 |
CRYOSPARC_CACHE_LOCK_STRATEGY | Distributed locking strategy to use when multiple running jobs access the SSD cache simultaneously. Set to file to use a file-system POSIX lock. Set to master to use the CryoSPARC master as a broker. Use the master strategy for caches on distributed file systems such as GPFS and BeeGFS where POSIX locks are disabled or unavailable. | master |
CRYOSPARC_CACHE_MAX_ATTEMPTS | Number of attempts to retry using the CryoSPARC cache. The job will fail if the number of attempts exceeds this value. | 1 |
CRYOSPARC_CACHE_NUM_THREADS | Number of threads to use during caching when copying particle .mrc files from the project directory to the cache directory. Set to 1 to disable threading and copy files sequentially. See /pages/-MNeFvdldyAikdOWQ4K3#leveraging-multiple-threads-to-copy-particles for details. | 2 |
CRYOSPARC_CLI_SKIP_ACCESS_CHECK | Set to 1 or true to disable filesystem permission checks for cryosparcm command-line arguments; may be used with network file systems where listed UNIX access permissions do not match the true available permissions. | false |
CRYOSPARC_IO_FD_LIMIT | Limit of how many open file descriptors CryoSPARC can create when running jobs. | - |
CRYOSPARC_IO_URING | Set to false to force-disable io_uring for fast disk read operations during jobs. May be required for some older operating systems or systems with an incorrect io_uring implementation. | true |
CRYOSPARC_NO_PAGELOCK | By default, CryoSPARC uses the CUDA driver's pagelocked_empty function to allocate GPU memory. This causes CUDA-driver errors on some systems. Set this variable to true to use numpy.empty | false |
CRYOSPARC_SSD_PATH | Set this variable to override the SSD cache path provided when you installed the worker. Useful if the SSD cache path is generated by your cluster as an environment variable when the job is scheduled. Important: The worker must be connected with a stub SSD path for this to take effect, e.g., "cache_path": "/tmp" in cluster_config.json | - |
](https://beta.cryosparc.com/docs/intro) # Guide: Installation Testing with cryosparcm test This guide covers how to use cryosparcm test to verify your CryoSPARC installation is working properly. {% hint style="warning" %} The information in this section applies to CryoSPARC v4.0+. {% endhint %} ## Overview After installing CryoSPARC [using the instructions here](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure), you can verify your instance is correctly installed by using `cryosparcm test install` and `cryosparcm test workers` via the command line. Running these functions will perform several tests that ensure users can seamlessly launch jobs and process data in CryoSPARC. ## `cryosparcm test install` This function tests the core components of CryoSPARC (HTTP connections, licensing, workers, etc.) that are required to start running jobs and provides information on the status of the CryoSPARC instance (e.g., which version is running, whether a patch is available, etc.). To run this function, log into a shell on the master node as the user that owns the CryoSPARC instance. Run `cryosparcm test -h` for full usage instructions. ### Example Output ``` cryosparcuser@uoft ~/ $ cryosparcm test i ✓ Running as CryoSPARC owner cryosparcuser ✓ Running on master node uoft ✓ CryoSPARC is running ✓ CRYOSPARC_LICENSE_ID environment variable is set ✓ Insecure mode is disabled ✓ License server set to "https://get.cryosparc.com" ✓ Connection to license server succeeded ✓ License server returned success status code 200 ✓ License server returned valid JSON response ✓ License exists and is valid ✓ CryoSPARC is running v5.0.0 ✓ Develop version - no updates available ✓ Admin user exists ✓ GPU worker connected ``` ### Test Checklist Running `cryosparcm test install` will test or check the following components: 1. Test if the `cryosparcm test install` command is running as the user who owns the CryoSPARC instance. 2. Test if the `cryosparcm test install` command is running on the machine that runs the CryoSPARC master instance. 3. Check if the CryoSPARC instance is turned on. * If this test fails, turn on CryoSPARC by running `cryosparcm start` and run the command again. 4. Test if an HTTP connection can be successfully created to the `api` (`CRYOSPARC_BASE_PORT`+2) server. * If this test fails, ensure a firewall isn’t blocking access to the ten consecutive ports from `CRYOSPARC_BASE_PORT` (default 61000, e.g., 61000-61010). For more information, see [Open TCP Ports](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites#4.-open-tcp-ports) in the Guide. 5. Check if the environment variable `CRYOSPARC_LICENSE_ID` is set. 6. Test if the CryoSPARC License ID is in the correct format. 1. If this test fails, ensure the CryoSPARC License ID found in `cryosparc_master/config.sh` is set to the correct license ID. 7. Check if insecure request mode is enabled or disabled. 1. This option is controlled by the `CRYOSPARC_INSECURE` environment variable found in `cryosparc_master/config.sh`. 2. Enabling this option ignores SSL certificate errors when connecting to HTTPS endpoints. This is useful if you are behind an enterprise network using SSL injection. 8. Check if the URL to the license server is valid. 1. The URL can be overridden by the `CRYOSPARC_LICENSE_SERVER_ADDR` environment variable found in `cryosparc_master/config.sh`. 2. The default URL is
You can find the jobs used for worker tests in the "Instance Testing Utilities" section of the job builder.
Workspace card of an instance testing run.
Switch the Extensive Workflow job from "Testing" mode to "Benchmark" mode using the drop-down menu.
Jobs run in Benchmark Mode (up to CryoSPARC v4.7)
1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Manually Curate Exposures 5. Blob Picker 6. Inspect Particle Picks 7. Extract From Micrographs (CPU) 8. 2D Classification 9. Select 2D 10. Template Picker 11. Inspect Particle Picks 12. Extract From Micrographs (CPU) 13. 2D Classification (50 Class) 14. 2D Classification (100 Class) (All Job Types Enabled) 15. 2D Classification (200 Class) (All Job Types Enabled) 16. Select 2D 17. Particle Sets Tools 18. Ab-Initio Reconstruction 19. Ab-Initio Reconstruction (3 Class) (All Job Types Enabled) 20. Homogeneous Refinement 21. Non-Uniform Refinement 22. Homogeneous Refinement Legacy (All Job Types Enabled) 23. Non-Uniform Refinement Legacy (All Job Types Enabled) 24. 3D Classification 25. 3D Variability (3 Mode) 26. 3D Variability (6 Mode) (All Job Types Enabled)Jobs run in Testing Mode (up to CryoSPARC v4.7)
1. Import Movies 2. Patch Motion Correction 3. Full-Frame Motion Correction 4. Patch CTF Estimation 5. CTFFIND4 6. Curate Exposures (Stream A) 7. Curate Exposures (Stream B) 8. Blob Picker (Stream A) 9. Blob Picker (Stream B) 10. Inspect Picks (Stream A) 11. Inspect Picks (Stream B) 12. Extract From Micrographs (Stream A) 13. Local Motion Correction (Stream B) 14. 2D Classification (Stream A) # all jobs past this point are in Stream A 15. Select 2D 16. Template Picker 17. Inspect Picks 18. Extract From Micrographs 19. 2D Classification (50 Class) 20. 2D Classification (100 Class) 21. 2D Classification (200 Class) 22. Select 2D 23. Particle Sets Tools 24. Ab-Initio Reconstruction 25. Ab-Initio Reconstruction (3 Class) 26. Homogeneous Refinement 27. Non-Uniform Refinement 28. Homogeneous Refinement (Legacy) 29. Non-Uniform Refinement (Legacy) 30. Heterogeneous Refinement (3 Class) 31. Heterogeneous Refinement (6 Class) 32. 3D Classification (Simple mode) 33. 3D Classification (PCA mode) 34. 3D Variability (3 mode) 35. 3D Variability (6 mode) 36. Sharpening Tools 37. Validation (FSC) 38. Global CTF Refinement 39. Local CTF Refinement 40. 3D Variability DisplayJobs run in Benchmark Mode (CryoSPARC v5.0+)
1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Manually Curate Exposures 5. Blob Picker 6. Inspect Particle Picks 7. Extract From Micrographs (CPU) 8. Micrograph Junk Detector (v5.0+) 9. 2D Classification 10. Downsample Particles (v5.0+) 11. Select 2D 12. Template Picker 13. Inspect Particle Picks 14. Extract From Micrographs (CPU) 15. 2D Classification (50 Class) 16. 2D Classification (100 Class) (All Job Types Enabled) 17. 2D Classification (200 Class) (All Job Types Enabled) 18. Select 2D 19. Particle Sets Tools 20. Ab-Initio Reconstruction 21. Ab-Initio Reconstruction (3 Class) (All Job Types Enabled) 22. Heterogeneous Refinement (v5.0+) 23. Homogeneous Refinement 24. Non-Uniform Refinement 25. Subset Particles by Statistic 26. Volume Tools 27. Orientation Diagnostics 28. 3D Variability (3 Mode) 29. 3D Variability (6 Mode) (All Job Types Enabled) 30. Local RefinementJobs run in Testing Mode (CryoSPARC v5.0+)
1. Import Movies 2. Import Micrographs 3. Import 3D Volumes 4. Import Particle Stack 5. Import Templates 6. Patch Motion Correction 7. Full-Frame Motion Correction 8. Import Beam Shift 9. Patch CTF Estimation 10. CTFFIND4 11. Check For Corrupt Micrographs 12. Generate Micrograph Thumbnails 13. Curate Exposures (Stream A) 14. Curate Exposures (Stream B) 15. Exposure Tools 16. Exposure Sets Tool (Split) 17. Exposure Sets Tool (Intersect) 18. Blob Picker (Stream A) 19. Blob Picker (Stream B) 20. Inspect Picks (Stream A) 21. Remove Duplicate Particles 22. Blob Picker Tuner 23. Inspect Picks (Stream B) 24. Reassign Particles to Micrographs 25. Patch CTF Extraction 26. Local Motion Correction (Stream B) 27. Local Motion Correction (Multi) (Stream B) 28. Extract From Micrographs (CPU) (Stream A) 29. Extract From Micrographs (GPU) 30. Micrograph Junk Detector 31. Restack Particles 32. 2D Classification (Stream A) # all jobs past this point are in Stream A 33. Cache Particles on SSD 34. Check For Corrupt Particles 35. Class Probability Filter 36. Downsample Particles 37. Select 2D 38. Average Power Spectra 39. Template Picker 40. Inspect Picks 41. Extract From Micrographs 42. Exposure Group Utilities (Combine Particles) 43. Exposure Group Utilities (Split Particles) 44. Exposure Group Utilities (Combine Exposures) 45. Exposure Group Utilities (Split Exposures) 46. 2D Classification (50 Class) 47. 2D Classification (100 Class) 48. 2D Classification (200 Class) 49. Rebalance 2D Classes 50. Select 2D 51. Particle Sets Tools 52. Reconstruct 2D Classes 53. Ab-Initio Reconstruction 54. Ab-Initio Reconstruction (3 Class) 55. Split Volumes Group 56. Simulate Data 57. Reference Based Auto Select 3D 58. Volume Tools 59. Create Templates (Non-Helical) 60. Create Templates (Helical) 61. Heterogeneous Refinement 62. Homogeneous Refinement 63. Non-Uniform Refinement 64. Subset Particles by Statistic 65. Volume Alignment Tools 66. Volume Tools (Mask) 67. Reference Based Auto Select 2D (Sobel) 68. Reference Based Auto Select 2D (Cluster) 69. Reference Based Auto Select 2D (Thresholds) 70. Rebalance Orientations 71. Orientation Diagnostics 72. Heterogenous Reconstruction Only 73. Local Resolution Estimation 74. Symmetry Search Utility 75. Apply Helical Symmetry 76. Reference Based Motion Correction 77. 3D Flex Data Prep 78. Patch Motion to Local Motion 79. 3D Classification 80. Sharpening Tools 81. Validation (FSC) 82. Global CTF Refinement 83. 3D Variability (3 mode) 84. 3D Variability (6 mode) 85. Local Refinement 86. Recenter Trajectories 87. Apply Trajectories 88. Local Filtering 89. 3D Flex Mesh Prep 90. Align 3D Maps 91. Regroup 3D Classes 92. Particle Subtraction 93. 3D Flex Training 94. Local CTF Refinement 95. ResLog Analysis 96. 3D Variability Display 97. 3D Flex Generator 98. 3D Flex ReconstructionJobs run in Benchmark Mode (up to CryoSPARC v4.7)
1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Curate Exposures 5. Blob Picker 6. Inspect Picks 7. Extract From Micrographs 8. 2D Classification 9. Select 2D 10. Template Picker 11. Inspect Picks 12. Extract From Micrographs 13. 2D Classification 14. Select 2D 15. Particle Sets Tools 16. Ab-Initio Reconstruction 17. Homogeneous Refinement 18. Non-Uniform Refinement 19. 3D Classification 20. 3D VariabilityJobs run in Testing Mode (up to CryoSPARC v4.7)
1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Filament Tracer 5. Inspect Picks 6. Extract From Micrographs 7. 2D Classification 8. Select 2D 9. Helical Refinement 10. Local CTF Refinement 11. Global CTF Refinement 12. Symmetry Expansion 13. Homogeneous Reconstruct OnlyJobs run in Benchmark Mode (CryoSPARC v5.0+)
1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Filament Tracer 5. Inspect Particle Picks 6. Extract From Micrographs (CPU) 7. 2D Classification 8. Select 2D Classes 9. Helical Refinement 10. Local CTF Refinement 11. Global CTF Refinement 12. Symmetry Expansion 13. Homogeneous Reconstruction OnlyJobs run in Testing Mode (CryoSPARC v5.0+)
1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Filament Tracer 5. Inspect Particle Picks 6. Extract From Micrographs (CPU) 7. 2D Classification 8. Select 2D Classes 9. Helical Refinement 10. Local CTF Refinement 11. Global CTF Refinement 12. Symmetry Expansion 13. Homogeneous Reconstruction Only| Tag Name | Most likely bottleneck |
|---|---|
| CPU Performance | - single core clock speed |
| GPU Performance | - float32 performance and memory bandwidth |
| PCIe Latency/Bandwidth Speed | - PCIe generation (3, 4) and number of lanes per slot (x8, x16) |
| GPU/CPU Memory Allocation | - general cpu/gpu performance |
| Input/Output Speed | - random read speeds of the storage device where particle images are located |
The Cluster Configuration tab of the Admin Panel
## Tips and Tricks ### Consolidating a Particle Stack When caching a particle stack that is larger than space available on your SSD, you may optionally consolidate your particle stack. This option works if the current particle stack is a subset of the original particle stack. For example, when the cache reports how much data it's requesting to copy (`SSD cache : cache requires 1000000.00 MB more on the SSD for files to be downloaded.` & `SSD cache : cache successfully requested to check 2000000 files.`) and the sizes it reports seem much larger than you expected, you can consolidate your particle stack such that only the particle subset you care about is cached. You might run into this situation if you ran an "Inspect Picks" job after an "Extract From Micrographs" job, and you modified the picking thresholds of your particles to include a smaller subset than the original stack. You might also run into this situation after a round of 2D Classification. When you select classes, you create metadata that specifies which subset of the particle stack to use. When using this particle subset in further processing, the caching system will require the entire stack of particles to be cached, even though only the smaller subset is required. To consolidate your particle stack, build a "Downsample Particles" job, connect your particles, and run the job. There is no need to change any parameters - nothing will change about your particle dataset except for the `.cs` metafile that will be recreated to reflect the smaller subset. You can use this smaller dataset to continue processing. ### Dynamic SSD Cache Paths On some systems it is not possible to know the SSD cache path ahead of time. Instead, a dynamically-generated path is available for jobs to use at run-time. To prompt a job to use this path, make the path available via a system-defined environment variable. Open `cryosparc_worker/config.sh` for editing and set the value of `CRYOSPARC_SSD_PATH` in the worker environment config to this variable: ``` # cryosparc_worker/config.sh export CRYOSPARC_LICENSE_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" export CRYOSPARC_USE_GPU=true export CRYOSPARC_CUDA_PATH="/usr/local/cuda" export CRYOSPARC_SSD_PATH="$CUSTOM_DYNAMIC_SSD_PATH" ``` ### Increase or Reduce Cache Files Lifetime As of CryoSPARC v3.3, jobs that require cache automatically remove cache files that have not been accessed in over 30 days. If projects on your instance get actively worked on for more or less time, you may change the cache file lifetime by adding the following line to `cryosparc_master/config.sh`: ``` export CRYOSPARC_SSD_CACHE_LIFETIME_DAYS=15 ``` Substitute `15` with the number of days your projects typically get worked on. ### Leveraging Multiple Threads to Copy Particles In CryoSPARC v4.3.0+, multiple threads (default 2) are used to copy particles from the project directory to the local cache device. To modify the number of threads used, add the following line to `cryosparc_worker/config.sh,` where `num_threads` is the number of threads (e.g., 12) to spawn to copy files: ``` export CRYOSPARC_CACHE_NUM_THREADS=num_threads ``` Specify `export CRYOSPARC_CACHE_NUM_THREADS=1` to turn off multithreading and copy particle files sequentially in the main process. ## Troubleshooting ### `SSD cache : cache waiting for requested files to become unlocked.` This temporary message usually means the files this job is trying to access are currently being cached by another job. For example, if you started two different Refinement jobs at the same time on the same node (Job A and Job B) using the same particle stack that haven't been cached on SSD before, both jobs try to first copy all particles onto the SSD. If Job A acquires the lock for the files first, it starts copying them and Job B shows this message. When Job A finishes copying the files, it unlocks them. Job B is unlocked and finds that the particles are already on the SSD, so it skips over the copy step. ### `SSD cache : cache does not have enough space for download... but there are no files that can be deleted.` This message means that there is another CryoSPARC job or another application on the workstation taking up space on the SSD. If the former, the job showing this message will try to free up space as soon as it can, and it will continue processing. If there are files on the SSD that are not owned by CryoSPARC, it will not be able to delete them. It may be necessary to delete them manually. ## FAQ **Is it safe to manually delete cache files for completed or unqueued/cleared jobs? Also, can I pre-cache with symlinks to skip caching?** Yes, it is safe to delete cache files any time (it’s a read-only cache) and yes, the cache checks to see if files exist just based on path/size/modification date so symlinks should cause it to skip. Though it may be easier to just set the SSD Cache parameter to False in each job that you queue up. Source: [How to clear the cache in v2?](https://discuss.cryosparc.com/t/how-to-clear-the-cache-in-v2/2161) # Guide: Data Management in CryoSPARC (v4.0+) An overview of all data management utilities and common use cases. {% hint style="warning" %} The information in this guide applies to CryoSPARC v4.0+. For information about managing project directories in older versions, see [Guide: Data Management in CryoSPARC (≤v3.3)](/guides-for-v3/tutorial-data-management-in-cryosparc) {% endhint %} {% hint style="danger" %} Do not remove from the filesystem any directory that is managed by an[ attached CryoSPARC project](#2.-attaching-detaching-archiving-and-unarchiving-projects). First, either * perform the *Detach Project* CryoSPARC GUI action for unwanted project(s) * or, to delete an unwanted project from the CryoSPARC GUI **and erase the project's data from disk**, perform the *Delete Project* GUI action {% endhint %} {% hint style="info" %} For additional data cleanup utilities available in v4.3+, please see: [Guide: Data Cleanup (v4.3+)](/setup-configuration-and-management/software-system-guides/guide-data-cleanup-v4.3) {% endhint %} ## Overview Single particle cryo-EM projects and labs continue to operate at increasing scales. In CryoSPARC v4.0, we introduce improved workflows and tools for dealing with archiving, transferring, exporting, and importing projects. Tools introduced in previous versions of CryoSPARC for exporting and importing individual jobs and individual results of various types (particle stacks, exposure stacks, volumes, etc.) remain available. In CryoSPARC v4.0, the most important changes that have been made are: * CryoSPARC projects are now explicitly **locked** (or **attached**) to a single CryoSPARC instance at a time. In previous versions, it was possible for a project directory to be imported into and accidentally modified by two instances at the same time, causing metadata corruption. Now, each project directory contains a lock file that marks the projects as in-use by (i.e., **attached to**) a particular instance. * CryoSPARC project directories are now named based on the project title at creation time, rather than a numeric project-UID (e.g., "P12"). Numeric UIDs are only used to refer to each project within a single instance. This serves to ensure that project directories are user-recognizable, and that the numeric UID is not retained when a project moves from one instance to another. * The life-cycle of a CryoSPARC project is now separated from the CryoSPARC instance(s) that interact with the project. The following lifecycle actions can be taken on a project: * **Create:** an instance can create a new project, and that project lives in a unique and self-contained project directory on disk. The project directory is created at creation time of the project. The project is attached to the instance that created it (and therefore there is a lock file present in the project directory). Within the instance to which the project is attached, the project has a unique numeric UID. * **Detach:** a user can opt to detach a project from the instance to which it is currently attached. This action ensures that no jobs or background processes are running in the project, and then removes the lock file from the project directory. In the UI, the project that was previously attached displays as "Detached" and can no longer be interacted with. * **Attach**: a project directory that has previously been detached (and therefore has no lock file present) can be attached to an instance. When attachment is performed, all the workspaces, jobs, and Live sessions within the project directory are imported into the attaching instance, and the project becomes usable within this instance and is given a new numeric UID. A lock file is written to the project directory. Attach takes the place of the previous **Import** action. * **Archive:** A project that is attached to an instance can be "archived" without detaching the project. This instructs CryoSPARC that the project directory is no longer available for reading and writing at it's current location, but will become available (possibly at a new location) at some future time. A user should archive a project before moving the project directory to a different location on disk, for example a different filesystem, a backup location, tape archive, or cold-storage. Archiving ensures that no jobs or background processes are running in the project and marks the project as archived, but does not remove the lock file from the project. Once archived, the project can still be browsed in the UI, but can not be modified. * **Unarchive:** A project that has been archived can be resurrected in the same instance from which it was archived. When unarchiving, the user is prompted to provide the (possibly changed) location of the project directory on disk. For example, a project can be archived and then the project directory moved to a cold unaccessible backup. Later, when needed, the project directory can be restored to an accessible filesystem, and the project can be unarchived pointing at the new project directory location. This makes the project available once again for further processing. * As a minor change, output files of CryoSPARC jobs that are stored in job directories no longer have the `cryosparc_PXX_` prefix, since the numeric project UID can change when a project moves from one instance to another. In order to retain the existing behaviour of the CryoSPARC UI and limit confusion between different files, when CryoSPARC results are downloaded through the browser in the UI, the prefix `cryosparc_PXX_` is added to the local filename of the download in the browser, using the then-current numeric project UID. These changes make it much simpler to perform the following actions: * **Detaching** a project from one CryoSPARC instance and **attaching** it to another instance * **Sending** a project initially started at a centralized facility to a user who is going to continue processing at home in their own instance, by detaching and attaching * **Archiving** a project to remote/slow storage for later retrieval and resurrection * **Copying** a project directory to make a complete clone * **Changing** the name or location of a project directory on disk, by archiving and unarchiving The following actions remain possible, with no change in behaviour in v4.0: * **Reducing** the disk space used by a project by removing intermediate results created by jobs * **Uploading** the final results of a CryoSPARC job to an online repository * **Advanced manipulation** of CryoSPARC metadata at a low level or programatically, by exporting a results (e.g., a particle stack), manually modifying the associated .cs files, and importing again The following sections describe specific aspects of data management in CryoSPARC in more detail. ## 1. CryoSPARC Projects and Project Directories ### Projects and "Continuous Export" CryoSPARC workflows are naturally divided into projects. Each project should contain the work and jobs for one or more related data collection sessions that are associated with a given sample/target. Project boundaries are strict, in the sense that files and results from one project cannot be directly used in another project. Project directories are self-contained, and all image processing data (except for imported raw data, see [#7.-imported-data-in-project-directories](#7.-imported-data-in-project-directories "mention")) pertaining to a project is written to the project directory. A project directory always contains all the information needed to define that project. The project directory is written to every time certain actions are taken within CryoSPARC, for example changing project, workspace, or job metadata (titles, descriptions, etc), and when jobs complete processing. This **"continuous export"** model ensures that at any time, a project directory is self-contained and if anything goes wrong with a CryoSPARC instance or database, the projects remain intact and up-to-date, without the user having to manually trigger an export action. As a safety feature, a project directory can, at any time, be transferred/renamed/copied and **attached** as a valid project in any (other) CryoSPARC instance that can read the files. **This is true even if the original CryoSPARC instance that created the project is no longer functional.** See [#use-case-rescuing-a-project-from-an-inoperable-instance](#use-case-rescuing-a-project-from-an-inoperable-instance "mention") for details on how to rescue a project from a failed or inoperable instance. Similar to projects, jobs inside the project are stored in a self-contained format, and job directories are updated whenever the jobs are created, modified, or completed. **Note:** jobs that are in `launched`, `started`, `running`, `waiting`, `killed` or `failed` status will not be updated on disk until they enter `completed` status - either by actually completing, or by the user choosing the `mark as completed` option in the Job Details panel. ### **Project directories and lock files** For projects created in CryoSPARC v4.0+, the project directory is initially named based on the title of the project at creation time. For example, if a project is titled\ "My Protein, Data Collection (October 1 2022)"\ the project directory will be created as\ `CS-my-protein-data-collection-october-1-2022`.\ The project directory will be created within the container directory indicated at creation time. The project directory can be changed later on (see [#use-case-renaming-a-project-directory](#use-case-renaming-a-project-directory "mention")). Inside each project directory in CryoSPARC v4.0+ (including existing projects), there will be a lock file present called `cs.lock`. **This file should not be removed or changed.** ### ## 2. Attaching, Detaching, Archiving, and Unarchiving Projects ### **Attach** Detached projects can be **attached** to a CryoSPARC instance. Attaching a project creates a new project in the instance using the indicated project directory. All project details, workspaces, jobs, and sessions in the detached project will be made accessible, and the project directory will be treated as an active project directory by the instance. Any intact CryoSPARC project directory that does not contain a lock file (including from a previous CryoSPARC version) can be attached. {% hint style="warning" %} Note: Projects already belonging to another CryoSPARC instance **cannot** be attached until they are detached from their original instance. When this is not possible see [#use-case-rescuing-a-project-from-an-inoperable-instance](#use-case-rescuing-a-project-from-an-inoperable-instance "mention") {% endhint %} Projects can be attached under the “New Project” dropdown menu:
Project-level defaults for generation of intermediate results
By default, none of the options will be selected. This allows for fine-tuning exactly what data should be cleared for a given project or workspace.
Right-click menu with option to mark job as final
Job marked as final
Instructions for CryoSPARC ≤ v3.3
If you're moving the locations of the projects and their jobs, you will need to point CryoSPARC to the new directory where the projects reside. Jobs inside CryoSPARC are referenced by their relative location to their project directory. This allows a user to specify a new location for the project directory only, rather than each job. **Update a Single Project** `cryosparcm cli "update_project('PXX', {'project_dir' : '/new/abs/path/PXX'})"` * Where `'PXX'` is the project UID and `'/new/abs/path/PXX'` is the new directory. **Updating Multiple Projects** **Step One - Identify All Project Directories** Start up an interactive python session ``` cryosparcm icli ``` Execute a MongoDB query to list all project directories ```python >>> projects = list(db['projects'].find({}, {'uid': 1, 'project_dir': 1, '_id': 0})) >>> projects [{'project_dir': '/data/cryosparc_projects/P1', 'uid': 'P1'}, {'project_dir': '/data/cryosparc_projects/P2', 'uid': 'P2'}, {'project_dir': '/data/cryosparc_projects/P3', 'uid': 'P3'}, {'project_dir': '/data/cryosparc_projects/P4', 'uid': 'P4'}, ...] ``` **Step Two - Modify One or Many Project Directories** Use the command `update_project(project_uid, attrs, operation='$set')` where `attrs` is a dictionary whose keys correspond to the fields in the project document to update. In the following example, `update_project` used in a loop to modify all project directory paths. ```python >>> failed_projects = [] >>> new_parent_dir = '/cryoem/cryosparc_projects' >>> for project in projects: new_project_dir = os.path.join(new_parent_dir, os.path.basename(project['project_dir'])) try: print("Modifying project directory for %s: %s --> %s" % (project['uid'], project['project_dir'], new_project_dir)) cli.update_project(project['uid'], {'project_dir': new_project_dir}) except Exception as e: failed_projects.append(project['uid']) print("Failed to update %s: %s" % (project['uid'], str(e))) ... >>> failed_projects [] ```## **10. Amazon FSx for Lustre** [Amazon FSx for Lustre](https://aws.amazon.com/fsx/lustre/?nc=sn\&loc=0) is a fully managed service that provides cost-effective, high-performance storage for compute workloads. Many workloads such as machine learning, high-performance computing (HPC), video rendering and financial simulations depend on compute instances accessing the same set of data through high-performance shared storage. Amazon FSx for Lustre file systems can also be linked to Amazon S3 buckets, enabling access and process data concurrently from a high-performance file system. Amazon FSx for Lustre can also be configured to back up data to Amazon S3, and further to Amazon S3 Glacier to optimize costs for data backup. You can choose between two file systems when using Amazon FSx for Lustre: 1. Persistent File Systems - these are ideal for long-term storage. 2. Scratch File Systems - these are ideal for temporary and short-term storage. This guide uses the Scratch file system. This provides the most performant and cost-effective option. If you require data resilience, consider the persistent deployment guide. ## **11. AWS CloudFormation** [AWS CloudFormation](https://aws.amazon.com/cloudformation/) provides a way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycles, by treating infrastructure as code. A CloudFormation template describes your desired resources and their dependencies so you can launch and configure them together as a stack. You can use a template to create, update and delete an entire stack as a single unit, as often as you need to, instead of managing resources individually. You can manage and provision stacks across multiple AWS accounts and AWS Regions. The script *`vpc-cryosparc.template`* is a CloudFormation template that deploys the VPC and subnets. As is, the script creates a VPC and two subnets. The subnet for the head instance is public, and the compute instances are placed in a private subnet. Outside the VPC, you can only log into the head instance in the public subnet (via SSH). ## **12. AWS ParallelCluster** [AWS ParallelCluster ](https://aws.amazon.com/hpc/parallelcluster/)enables you to quickly build an HPC environment on AWS. It automatically sets up the required compute resources, a shared filesystem, and offers a variety of batch schedulers. You define all the resources you need in a config file. This deployment will use ParallelCluster 2.10.1. CryoSPARC requires the multiple queues feature in ParallelCluster, which is only supported by versions 2.9.0 and later. To install ParallelCluster 2.10.1, run the following command on your local machine: ``` $ pip3 install aws-parallelcluster --user ``` This enables access to the terminal command-line tool `pcluster`. First, confirm that you have the correct version installed. ``` $ pcluster version 2.10.1 ``` ### **12.1. Configure the cluster** Open *`cryosparc-pcluster.config.template`* in a text editor; here, provide the details of the cluster required to deploy and run the CryoSPARC workflow. An AWS ParallelCluster configuration is defined in multiple sections. A section starts with the section name in square brackets, followed by parameters and configuration. [See this page](https://docs.aws.amazon.com/parallelcluster/latest/ug/configuration.html) for more information about the config file. At the time of deployment, a *`cryosparc.config`* file is created from the *`cryosparc.config.template`* file (more on this later). Look through the values that are not explicitly defined in *`cryosparc-pcluster.config.template`*. * `aws_region_name` - the name of the region * `key_name` - key pair name * `post_install` - the path to the install script that runs after the cluster is created * `s3_read_resource` - the S3 bucket with raw data * `vpc_id` - the VPC id * `master_subnet_id` - the subnet id where the head node resides * `compute_subnet_id` - the subnet id where the compute node resides You will provide the region, key pair name S3 bucket names and CryoSPARC license ID when you deploy the cluster. The *`deploy-cryosparc.sh`* fetches the remaining values about the networking infrastructure created by *`vpc-cryosparc.template`*. This config file uses an Amazon EC2 c5n.9xlarge instance as the head node. The c5n.9xlarge instance uses an Intel Xeon Platinum (Skylake) processor with 36 vCPUs. All instances use Amazon Linux 2, a Linux server operating system used by Amazon Web Services (AWS), as their operating system. None of the compute instances are running yet; they will run as needed. Three computing queues are also defined: **gpu-large**, **gpu-med**, and **gpu-small**. Each queue can host multiple EC2 instances. For example, the **gpu-large** queue is made of p4d.24xlarge, p3.16xlarge, and g4dn.metal instances. Multiple queues are very useful since different steps of the CryoSPARC workflow run better with different instances. See the attached benchmarking guide for instance recommendations. ### **12.2. How to deploy** In your local directory, verify that the three provided scripts exist: * *`cryosparc-pcluster.config.template`* * *`deploy-cryosparc.sh`* * *`vpc-cryosparc.template`* Before launching the cluster, choose the Availability Zone (AZ) where the cluster will run. All EC2 instance types you choose to instantiate should be available within the same AZ in the region your S3 bucket is in. If a required EC2 instance type is not available in any of the AZs in your region, re-create your S3 bucket in a region that has them available. Run the command below to see where a given instance type is available. ``` $ aws ec2 describe-instance-type-offerings \ --location-type availability-zone \ --region your-aws-region \ --filters Name=instance-type,Values=type of your instance \ --query "InstanceTypeOfferings[*].Location" \ --output text ``` * `--region` Region you want to deploy the cluster in * `Values=` type of instance For example - this command outputs the AZs in the us-east-1 region where g4dn.12xlarge instances are available. ``` $ aws ec2 describe-instance-type-offerings \ --location-type availability-zone \ --region us-east-1 \ --filters Name=instance-type,Values=g4dn.12xlarge \ --query "InstanceTypeOfferings[*].Location" \ --output text us-east-1b us-east-1f us-east-1d us-east-1c us-east-1a ``` Finally, to deploy the cluster ``` $ ./deploy-cryosparc_v1.sh --region your-aws-region \ --cluster-name cryosparc \ --az
### **Cost Analysis** Using a single EC2 instance type for an entire CryoSPARC workload is not an optimal solution. Instances like the p4d.24xlarge will provide the best performance, but with no consideration to cost. General guidelines were given above as to what instances to pick for what stage. Below is an example of how a user may implement those recommendations, along with the total cost (including storage), and runtime. Prices are listed in USD for the us-east-1 region. | | **Configuration 1** | | **Configuration 2** | | | | | -------------------------------------------------------------------------- | ------------------- | -------------- | ---------------------------------------------------------------- | ----------------- | -------------- | --------------------------------------------------------------- | | **Pipeline Stage** | **Instance Type** | **Cost (USD)** | **Runtime (Min.)** | **Instance Type** | **Cost (USD)** | **Runtime (Min.)** | | **Patch Motion Correction** | p4d.24xlarge | $12.25 | 22.4 | g4dn.metal | $3.97 | 30.5 | | **Patch CTF Estimation** | p4d.24xlarge | $7.46 | 13.7 | g4dn.metal | $1.44 | 11.1 | | **Blob picker** | g4dn.16xlarge | $0.72 | 9.9 | g4dn.16xlarge | $0.72 | 9.9 | | **Template Picker** | g4dn.16xlarge | $1.77 | 24.4 | g4dn.16xlarge | $1.77 | 24.4 | | **Micrograph Extraction** | p3.8xlarge | $1.65 | 8.25 | g4dn.16xlarge | $0.90 | 12.5 | | **2D Classification** | g4dn.metal | $1.76 | 13.5 | g4dn.metal | $1.76 | 13.5 | | **Ab-initio Reconstruction** | g4dn.16xlarge | $2.56 | 35.3 | p3.16xlarge | $6.83 | 23.3 | | **Non-Uniform Refinement** | g4dn.16xlarge | $9.43 | 130 | p4d.24xlarge | $33.66 | 82.5 | | **Compute Cost** | | $37.60 | | | $51.05 | | |
Lustre Cost
(12 TB Scratch)
| | $9.52 | | | $8.03 | | | **Total** | | **$47.12** |257.55
(4.29 hrs)
| | **$59.08** |206.8
(3.45 hrs)
| For comparison, here are costs and runtimes if we ran the entire benchmark on a single EC2 instance type: * p4d.24xlarge * **2.79 hours** * **$91.40** * g4dn.metal * **4.8 hours** * **$37.60** * p3.16xlarge * **3.4 hours** * **$83.00** ## **Conclusion** The benchmark results presented demonstrate how AWS can be used to accelerate cryo-EM workloads. By making use of AWS ParallelCluster, users can easily create an HPC cluster with a range of GPU instance types, allowing them to best match compute resources with the requirements of each analysis step. Benchmark data presented here was done with a single user; in practice, it’s likely multiple users will use the cluster for analysis. Further cost optimization can be achieved by running multiple jobs on a larger instance. For example, while a p4d.24xlarge (with 8 A100 GPUs) may have a higher cost, running multiple, single-GPU stages like Non-Uniform Refinement at the same time will help amortize the higher cost of the p4d instance. Summarized below are the key points a user should consider when creating a cryoEM cluster. * Amazon FSx for Lustre provides a high-performance file system that can meet the requirements of a cryo-EM analysis pipeline. This also allows flexibility in choosing instance types (e.g. those that lack local, fast NVMe storage). * A range of GPU instances should be employed. AWS ParallelCluster can be used to easily create different queues with different EC2 instances for this purpose. * p3dn.24xlarge instances are not recommended. They can provide excellent performance, but the p4d.24xlarge instance is priced very closely to the p3dn.24xlarge. The time-to-solution for the p4d.24xlarge is fast enough that in most cases a processing stage will use less compute time and, thus, cost less. * g4dn instances will likely make up the bulk of the compute resources. They provide performance at an excellent price point. # Troubleshooting Overview of common issues and advice on how to resolve them. {% hint style="info" %} Unless otherwise noted: * Log in to the workstation or remote node where `cryosparc_master` is installed. * Use the same non-root UNIX user account that runs the CryoSPARC process and was used to install CryoSPARC. * Run all commands on this page in a terminal running `bash` \ In v4.0+, you can download error reporting information from within the application. For more details, see: [Guide: Download Error Reports](/setup-configuration-and-management/software-system-guides/guide-download-error-reports) {% endhint %} ## Common Issues ### Cannot download or install CryoSPARC Problems with the [**installation**](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc) steps are indicated with the some of the following error messages: > "Couldn't connect to host"\ > "Could not resolve host"\ > {"success": false}\ > "tar: This does not look like a tar archive"\ > "Version mismatch! Worker and master versions are not the same. Please update."\ > "An unexpected error has occurred." **Steps** 1. If you have [Conda](https://docs.conda.io/projects/conda/en/latest/index.html) installed, [deactivate any active environments](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#deactivating-an-environment). 2. Check that your `LICENSE_ID` environment variable is set correctly with this command ```bash echo $LICENSE_ID ``` Ensure the output exactly matches the CryoSPARC License ID issued to you over email. 3. Check your machine's connection to CryoSPARC's license servers at get.cryosparc.com with this `curl` command: ```bash curl https://get.cryosparc.com/checklicenseexists/$LICENSE_ID ``` \ You should see the message `{"success": true}`. If instead you see `{"success": false}`, your license is not valid, so please check it has been entered correctly.\ \ If you see an error message like **"Couldn't connect to host"** or **"Could not resolve host"** check your Internet connection, firewall or ensure your IT department allows access to the `get.cryosparc.com` license server domain. ### Cannot update CryoSPARC **Steps** * [Reinstall CryoSPARC](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc), following the steps from the ["Cannot download CryoSPARC"](#cannot-download-cryosparc) section ### CryoSPARC does not start or encounters error on startup This can happen following a fresh install or recent update. **Steps** 1. In a command line, run `cryosparcm status` 2. Check that the output looks like this ``` CryoSPARC System master node installed at /home/cryosparcuser/cryosparc/cryosparc_master Current CryoSPARC version: v5.0.0 -------------------------------------------------------------------------------- CryoSPARC process status: api:0 RUNNING pid 2439162, uptime 27 days, 19:52:14 api:1 RUNNING pid 2439163, uptime 27 days, 19:52:14 api:2 RUNNING pid 2439164, uptime 27 days, 19:52:14 app RUNNING pid 2443242, uptime 27 days, 19:52:04 app_api RUNNING pid 2443078, uptime 27 days, 19:52:06 cache RUNNING pid 2435460, uptime 27 days, 19:52:36 command_vis RUNNING pid 2440324, uptime 27 days, 19:52:08 database RUNNING pid 2435129, uptime 27 days, 19:52:38 scheduler RUNNING pid 2439743, uptime 27 days, 19:52:10 search RUNNING pid 2440388, uptime 27 days, 19:52:07 -------------------------------------------------------------------------------- ✓ License is valid -------------------------------------------------------------------------------- export CRYOSPARC_LICENSE_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" export CRYOSPARC_MASTER_HOSTNAME="localhost" export CRYOSPARC_DB_PATH="/home/cryosparcuser/cryosparc/cryosparc_database" export CRYOSPARC_BASE_PORT=61000 export CRYOSPARC_INSECURE=false export CRYOSPARC_CLICK_WRAP=true ``` 3. Check that all items under "CryoSPARC process status" that *do not* end in `_dev` or `_legacy` are `RUNNING`. If any are not, run `cryosparcm restart` 4. If any non-`_dev` /non-`_legacy` components have a status other than `RUNNING` (such as `STOPPED` or `EXITED`), check their log for errors. For example, this command checks for errors on the `database` process: ```bash cryosparcm log database ``` (Press `control C`, then `q` to stop logging) 5. If the web interface is inaccessible, check firewall settings to ensure [CryoSPARC's base port number](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#optional-re-load-your-bashrc-this-will-allow-you-to-run-the-cryosparcm-management-script-from-anywhere-in-the-system) is exposed for network access Any error messages here could indicate specific configuration issues and may require [**re-installing CryoSPARC**](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc). If at any point you see `No command 'cryosparcm' found` or `command not found: cryosparcm`: 1. Check that you are on the master node or workstation where `cryosparc_master` is installed 2. Run `echo $PATH` and check that it contains `4. Check the Event log: Select the Job card in the CryoSPARC interface and press the Spacebar on your keyboard to see the log. Scroll down to the bottom and look for the failure reasons in red\  5. Clear the job and select the "Build" status badge on the job card to enter the Job Builder 6. If the job failed with a GPU-related error and multiple GPUs are available, try running the job on a different GPU. Press Queue, switch to the "Run on specific GPU" Queue type and select one or more GPUs\  7. If the job failed with `AssertionError: Non-optional inputs from the following input groups and their slots are not connected` then clear the job, enter the Builder and expand any input groups connected to the job. Missing required slots appear with text "Empty" and "Required"\  8. Check the job parameters: To learn about setting specific parameters, hover over or touch the parameter names in the Job Builder to see a description of what they do  9. Find the target job type in this guide's Job Reference for more detailed descriptions of expected input slots and parameters: [All Job Types in CryoSPARC](/processing-data/all-job-types-in-cryosparc). 10. Reduce the box-size of extracted particles. Some jobs need to fit thousands of particles in GPU memory at a time, and larger box sizes exceed GPU memory limits. Either extract with a smaller box size or with the [Downsample Particles job](/processing-data/all-job-types-in-cryosparc/extraction/job-downsample-particles). 11. Look for extended error information with the [`cryosparcm job log` command](/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcm-reference-v5.0#cryosparcm-job-log) (press `Ctrl + C` on the keyboard to exit when finished) 12. Check the network connection from the worker machine to the master 13. On occasion, a job fails due to an error in the CryoSPARC code (bug). The CryoSPARC team regularly releases updates and patches with bug fixes. [Check for](https://cryosparc.com/updates) and [install the latest update](/setup-configuration-and-management/software-updates) or [patch](/setup-configuration-and-management/software-updates#patches).\ \ If you find a new bug, see the [Additional Help](#additional-help) section for advice. ### Job stuck or taking a very long time Due to their large sizes, cryo-EM datasets can take a long time to process with sub-optimal hardware or parameters. Here are some facilities that CryoSPARC provides for increasing speed/performance. * [Connect workers with SSD cache enabled](/setup-configuration-and-management/management-and-monitoring-4.7/cryosparcw-4.7#cryosparcw-connect). This speeds up processing for extracted particles during 2D Classification, *ab-initio* reconstruction, refinement and more. Ensure the "**Cache particle images on SSD**" parameter is enabled under **"Compute settings"** for the target particle-processing job * Some jobs (motion correction, ctf estimation, 2D classification) can run on multiple GPUs. If your hardware supports it, increase the number of GPUs to parallelize over  * Extracted particles with large box sizes (relative to their pixel size) take a long time to process. Consider Fourier-cropping (or "binning") the extracted particle blobs with the Downsample Particles job {% content-ref url="/pages/17vxAkUji9lhwxmzL6P5" %} [Job: Downsample Particles](/processing-data/all-job-types-in-cryosparc/extraction/job-downsample-particles) {% endcontent-ref %} * Minimize the number of processes using system resources on the workstation or worker nodes * Check for zombie processes on worker machines. The process is similar to the steps under **"Another CryoSPARC instance is running with the same license ID"** under the [License error or license not found](#license-error-or-license-not-found) section * Cancel the job, clear and re-queue ### GPU Issues #### `cudaErrorInsufficientDriver` or `CUDA_ERROR_UNSUPPORTED_PTX_VERSION` A job fails with errors similar to the following when the Nvidia driver is out-of-date or incompatible with the target GPU or CUDA Toolkit Version that ships with CryoSPARC: ```python File "/u/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1711, in run_with_except_hook run_old(*args, **kw) File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 129, in cryosparc_compute.engine.cuda_core.GPUThread.run File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 997, in cryosparc_compute.engine.engine.process.work File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 106, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu File "cryosparc_worker/cryosparc_compute/engine/gfourier.py", line 33, in cryosparc_compute.engine.gfourier.fft2_on_gpu_inplace File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/fft.py", line 102, in __init__ capability = misc.get_compute_capability(misc.get_current_device()) File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/misc.py", line 254, in get_current_device return drv.Device(cuda.cudaGetDevice()) File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cudart.py", line 767, in cudaGetDevice cudaCheckStatus(status) File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cudart.py", line 565, in cudaCheckStatus raise e skcuda.cudart.cudaErrorInsufficientDriver ``` ```python Traceback (most recent call last): driver.cuLinkAddData(self.handle, input_ptx, ptx, len(ptx), File "/home/cryosparcuser/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 352, in safe_cuda_api_call return self._check_cuda_python_error(fname, libfn(*args)) File "/home/cryosparcuser/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 412, in _check_cuda_python_error raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [CUresult.CUDA_ERROR_UNSUPPORTED_PTX_VERSION] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cryosparc_master/cryosparc_compute/run.py", line 96, in cryosparc_master.cryosparc_compute.run.main File "/home/cryosparcuser/cryosparc_worker/cryosparc_compute/jobs/instance_testing/run.py", line 174, in run_gpu_job func = mod.get_function("add") File "/home/cryosparcuser/cryosparc_worker/cryosparc_compute/gpu/compiler.py", line 256, in get_function cufunc = self.get_module().get_function(name) File "/home/cryosparcuser/cryosparc_worker/cryosparc_compute/gpu/compiler.py", line 212, in get_module linker.add_cu(s, k) File "/home/cryosparcuser/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 3022, in add_cu self.add_ptx(program.ptx, ptx_name) File "/home/cryosparcuser/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 3010, in add_ptx raise LinkerError("%s\n%s" % (e, self.error_log)) numba.cuda.cudadrv.driver.LinkerError: [CUresult.CUDA_ERROR_UNSUPPORTED_PTX_VERSION] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION ptxas application ptx input, line 9; fatal : Unsupported .version 7.8; current version is '7.3' ``` To fix this, update the Nvidia driver to the minimum driver version noted in [Installation Prerequistes](/setup-configuration-and-management/cryosparc-installation-prerequisites). Please follow instructions specific to the worker's Linux distribution to install the Nvidia driver. The latest Nvidia driver is available to download on [Nvidia's website](https://www.nvidia.com/Download/index.aspx). [Related Discussion Forum post where a user encountered this error.](https://discuss.cryosparc.com/t/cryosparc-tries-to-run-worker-in-old-directory-cannot-find/5632/5) #### **`undefined symbol: _ZSt28__throw_bad_array_new_lengthv`** A job may fail with the following error when running GPU jobs (Patch Motion Correction, 2D Classification) with CryoSPARC v3.3.2 or v3.4.0 on Ubuntu 22+. ```python Traceback (most recent call last): File "cryosparc_worker/cryosparc_compute/run.py", line 72, in cryosparc_compute.run.main File "/home/cryosparc/cryosparc/cryosparc_worker/cryosparc_compute/jobs/jobregister.py", line 371, in get_run_function runmod = importlib.import_module(".."+modname, __name__) File "/home/cryosparc/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "
db.cache_files.find({status: {$nin: ['hit', 'miss']}}).length()
db.cache_files.deleteMany({})
# Jobs The fundamental unit of data processing within CryoSPARC is a job. Jobs are given inputs and parameters, do work, and through that work, create outputs. The outputs of one job can be connected as the inputs of another job, and through this process, allow you to create branching pipelines through which you can explore and refine your data. ## Job Card The most common representation of a job in CryoSPARC is a “Job Card”. This card is composed of a number of different sections used to organize job information in a consistent way with flexibility to accommodate the specific differences between job types.
A 2D Classification job in the default image view
A 2D Classification job in the “outputs” view. Outputs are selectable for use in the "Job Builder" and “Job Cart”.
Missing connection error and resolution when creating a group
Missing connection error and resolution when adding jobs to an existing group
The list of jobs reduced to only those which can run on multiple GPUs
A list of job cards with the outputs mode active. All output groups will be listed alongside their item count. Clicking any group will add it to the cart.
With an output group added to the cart, the list of applicable jobs is automatically filtered. You can further filter this list via the search input and hover over any job to view additional details. Once you click to select or press enter, the job will be created and all groups within the cart will automatically be connected to the new job.
Right-clicking on a 2D Classification job presents two quick actions. Clicking ‘Queue Select 2D Classes’ will automatically create and queue this interactive job.
Right-clicking a multi-class Ab-initio Reconstruction presents an option to build a refinement job for each class that was reconstructed.
The job builder sidebar tab active with an Inspect Particle Picks job in building status
The Queue Slide-over in CryoSPARC v4.0
The Queue Slide-over in CryoSPARC v5.0+
The Queue Job sub-menu in CryoSPARC v4.0
The Queue Job sub-menu in CryoSPARC v5.0+
Selecting a single job will display its details in the sidebar ‘Details’ panel. You can press Spacebar, ‘View Job’ or click the job ID/type on the header of the card.
Inspecting a completed 2D Classification job presents the Event Log tab for quick inspection of the latest events.
You can close the job inspection dialog by clicking on the ‘X’ button on the top-right of the dialog or press the Spacebar or Escape key
The job dialog footer contains metadata with additional details visible when hovered.
Viewing a selection of related jobs connected to a Homogeneous Refinement
Each related job will display key information about its configuration and outputs
What is a passthrough?
In the figure above, you may note that the particles’ location and pick stats are *passthroughs*. When jobs do not modify the information in an output group, they are treated as passthroughs. Under the hood, this means that CryoSPARC does not save a new copy of the passed-through information for the new job. Instead, the job’s output contains the location of the original information. More information about passthroughs is available [here](https://guide.cryosparc.com/guides-for-v3/job-builder-tutorial#passthrough-results). CryoSPARC and CryoSPARC Tools both automatically fetch passthrough information, so in most cases you can treat passthroughs as if they were a typical output group slot. External programs which directly read `.cs` files, however, will not have access to passthroughs without special attention. To create a `.cs` file which contains all of the metadata (including passthroughs), [export](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-data-management-in-cryosparc#use-case-share-a-particular-job-with-another-user) the job first.View options panel for project cards
View options panel for project table
The Job History view in the flattened jobs view with applicable filters applied.
Creating a workflow
Tree view containing three applied workflows
Workflow job parameter
In v5.0+, lanes and individual jobs display up-to-date resource allocation data
An Exposure Curation job in building status with an auto-threshold on CTF Fit Resolution (Å) set. When launched with this configuration, the job will output only exposures within a CTF Fit range of 0-4Å.
How does the file upload work?
Individual files are split into 5 MB “chunks”, which are individually uploaded. During the upload process, the `uploads` directory will fill with these chunk files. Once the upload process is done, the chunks are combined back into the original file and removed. If the upload process is interrupted, the chunks generated before the failure will be orphaned in the filesystem. They will not interfere with any subsequent uploads, and will be automatically removed if the file is successfully transferred later. It is harmless to remove these orphaned chunks.| Item | File type | |
|---|---|---|
| 3D Volumes | .mrc, .map, .ccp4 | |
| Cryo-EM movies and micrographs | .eer, .mrc, .tiff, .tif | |
| Cryo-EM particle stacks | .mrc, .mrcs | |
| CryoSPARC metadata files | .cs | |
| BILD files | .bild | |
| Segmentation files | .seg | |
| JSON files | .json | |
| PDFs | .pdf | |
| Text files | .txt | |
| Compressed files | .tar, .tar.bz2, .tar.gz, .zip | |
| Images and videos | Any file with the MIME type image or video, including but not limited to .jpeg, .png, .mp4 |
CSV download button is located in the footer of all browse pages
Dialog displaying options to customize the CSV file
Viewing and downloading output results from the job inspection dialog.
Viewing and downloading outputs from the job details sidebar.
Downloading the alignments3D low-level result from iteration 2 of this Homogeneous Refinement job.
Instance Tab in v4.0
Instance Tab (Card View) in v5.0+
Instance Tab (Table View) in v5.0+
This object is C5 symmetric. A single ASU is highlighted. It can be rotated by 1/5 turn about the Z axis and remain unchanged. It has no other symmetry operations.
This object is D5 symmetric. A single ASU is highlighted. It can be rotated by 1/5 turn about the Z axis and/or a 1/2 turn about the Y axis and remain unchanged. It has no other symmetry operations.
This object is T symmetric. Note that the base is triangular. A single ASU is highlighted. The 3-fold axes pass through each vertex and the center of the opposite face, while the 2-fold axes pass through the centers of each opposing pair of edges.
This object is O symmetric. A single ASU is highlighted. The 4-fold axes are aligned to X, Y, and Z. The 3-fold axes pass through the centers of each opposing pair of faces. The 2-fold axes pass through the centers of each opposing pair of edges.
These objects have icosahedral symmetry. A single ASU is highlighted. The left object is aligned according to the I or I1 conventions, while the object on the right is aligned to the I2 convention. The 5-fold axes pass through opposing pairs of vertices. The 3-fold axes pass through the centers of opposing pairs of faces. The 2-fold axes pass through the centers of opposing pairs of edges.
The Fourier shell currently evaluated. Center: The real space signal corresponding to the current Fourier shell. Right: A plot of each shell’s correlation.
Why is the resolution cutoff 0.143?
The argument for this threshold was made in a seminal paper by Rosenthal and Henderson (2003). Here we will cover the basic outline of the justification of 0.143 as the resolution cutoff. The details and mathematics can be found in the original paper and are left to the interested reader. Note first that, ideally, the GSFSC would tell us something about the signal-to-noise ratio of our reconstruction. That is, we can say something like “If the GSFSC resolution is 3 Å, features in the map which are 3 Å and coarser are at least half real signal”. When the FSC is 0.5, the power in the map is half signal and half noise, so one might select a threshold of 0.5. However, the GSFSC is comparing two maps which are each made from only half the data, but we care about the signal content of the final, averaged map compared to an ideal map with no noise at all. A threshold of 0.5 for the GSFSC is therefore an underestimate of the information content in the final, averaged map. If we select a threshold of 0.143 for the FSC between two half maps, the correlation between the final, averaged map and the unknown perfect map is 0.5.The region outside the mask (i.e., where the mask is set to 0.0) is hatched with white.
In this example, phases are randomized starting with the unmasked GSFSC resolution (dotted vertical line). Note that, in Fourier space (blue), the maps become random noise starting at the corresponding distance from the origin.
| 2D Classification | E-M is used to find the most likely pose and class for each particle. Pose marginalization is controlled by Force max over poses/shifts, while class marginalization can be turned off for the final Maximization step by turning on Hard classify for last iteration. |
| Homogeneous Refinement and Non-Uniform Refinement | E-M is used to find the most likely pose for each particle. Pose marginalization is controlled by the Adaptive Marginalization parameter. Adaptive Marginalization is off by default in Homogeneous Refinement, but on by default in Non-Uniform Refinement. |
| Heterogeneous Refinement | E-M is used to find the most likely pose and class for each particle. Class marginalization is controlled by the Force hard classification parameter (off by default). Poses are always maximized. |
| Heterogeneous Reconstruction Only | Poses and class probabilities are fixed in this job, but class membership is marginalized using existing probabilities unless Force hard classification is turned on. |
| 3D Classification | E-M is used to find the most likely class for each particle (poses are fixed). Class membership is marginalized unless Force hard classification is turned on. Class similarity provides an additional control over how particles are marginalized over class membership. |
| Local Refinement | E-M is used to find the most likely pose for each particle. Poses are marginalized if Marginalization is turned on. Additionally, the likelihood of each pose can be fine-tuned with a prior probability if Use pose/shift gaussian prior during alignment is turned on. |
| Helical Refinement | E-M is used to find the pose for each image. No marginalization is performed. |
The CryoSPARC dashboard
Parameters with default values are marked in gray. Custom values are marked in green.
The job queueing dialog displaying a list of lanes (configured via the CryoSPARC command-line interface) to choose from. Here, we have chosen to queue the job to the cryoem2 lane.
The blue icon next to the job ID (J2) indicates this job is running.
Connecting Patch Motion Correction job outputs to a building Patch CTF Estimation job and queuing it to a GPU-accelerated instance.
Comparison between simple lowpass filtering (left) and denoising (right) of the same micrograph.
Example of building a Blob Picker job from a Micrograph Denoiser job, using Quick Actions.
A portion of the Blob Picker event log depicting the template the algorithm uses for picking and exposure image with pick selections plotted as magenta squares.
Job details dialog for the Blob Picker open while the Inspect Particle Picks job details panel is active in build mode.
Output result groups of the Inspect Picks job showing the resulting 20 micrographs and 12,700 selected particles.
Workspace with the Extract from Micrographs job builder active.
Output result groups of the Extract from Micrographs job showing the resulting 20 micrographs and 11,129 extracted particles.
Generated 2D classes from extracted blob picks.
Interactive Select 2D job depicting two classes selected representing primary orientations.
Event log of the Select 2D job depicting the two classes selected and 48 classes excluded.
Job details dialog depicting the completed Template Picker job with 18,410 picked particles.
Job details dialog depicting the completed Inspect Picks job with 13,808 resulting particles.
Output result groups of the Extract from Micrographs job showing the resulting 20 micrographs and 12,501 extracted particles.
Generated 2D classes from the template picked particles.
Interactive Select 2D job depicting good quality classes selected.
20 selected classes and 30 excluded classes.
Event log of the completed Ab-initio Reconstruction job.
Event log of the completed Homogeneous Refinement job.
Gold Standard Fourier Shell Correlation (GSFSC) plot for the final refinement iteration.
Within the job details dialog, output groups listed have a download menu with various options.
The sharpened map (blue) is displayed with the lowpass filtered map (orange) at various thresholds. The threshold on the left is too high, resulting in too much of the underlying map to be outside the mask. The threshold on the right is too low, resulting in noise in the mask. The center threshold is good, because it preserves all of the information in the map while excluding noise.
The event log of a completed Validation (FSC) job.
A comparison of the Corrected and Tight FSC curves produced by Homogeneous Refinement (top) and Validation FSC (bottom). In this case, the two curves are very similar. However, the FSC calculated by Homogeneous Refinement has a dip away from the Tight curve, indicating that the mask may have been slightly too tight.
Guinier plot from the final refinement iteration depicting B-Factor value.
Tree view of the full T20S processing workflow.
What is a gain reference?
Gain references are used to account for differences in the ability of each pixel to detect an electron. A given pixel may be more or less sensitive to electrons than its neighbors. This difference in sensitivity can lead to banding patterns or other artifacts that do not reflect the true sample image. Gain references are special images (typically recorded in a manufacturer-specific format and converted to `.mrc`) which are multiplied by the experimental image to correct for these artifacts.An example thumbnail from an Import Movies job. On the left, the gain reference has not been flipped, yielding thumbnails with large dark and light stripes. Turning on “Flip gain ref & defect file in Y?” produced the image on the right. Data from EMPIAR 10288 (Kumar et al. 2019).
The export button can be found on the left-hand side of each result group’s box in the Outputs tab of a completed job.
In this movie of intact SARS-CoV-2 virions (EMPIAR 10492), the particles in total move only approximately 10 Å, but loss of contrast due to motion blurring is still clear. Motion correction recovers signal lost due to motion blurring, increasing contrast and improving the quality of the final results. Though not obvious in individual micrographs, the effect of motion correction is most important in improving high-resolution signal quality.
A comparison of a map made from the first fifteen or last sixteen frames of a set of movies. The same particles are used in the same poses in both images — only the frames used during motioncorrection differ. The data are from EMPIAR 10424 (Nakane et al. 2020). The EER movies were fractionated into 31 frames with an upsampling factor of 2. The same 500 movies were used in each patch motion correction job. The motion correction jobs were run with default parameters, except F-crop set to 1/2. Published poses were used for homogeneous reconstruction of particles extracted from non-dose-weighted micrographs at a box size of 700 px, downsampled to 320 px.
An example micrograph from EMPIAR 10674.
An example of a single frame’s patch displacement. An arrow represents the displacement of a patch in a single frame. Note that the length of the arrows are scaled for visibility.
X and Y displacement models for a single frame are shown with pink and blue surfaces, respectively. Using this function, each pixel’s displacement can be modeled.
Each pixel’s displacement varies as it moves through time, and so do the spline functions.
A schematic overview of a micrograph showing particle locations and trajectories (axis labels are in pixels)
Example motion-corrected particles.
An example of realistic trajectories. Each black dot is a particle pick location, and the tail associated with it is the motion estimate for that particle over the length of the exposure.
Class 1, patch motion correction
Class 2, patch motion correction
Class 1, reference-based motion correction
Class 2, reference-based motion correction
A micrograph from EMPIAR-10424 (Nakane et al. 2020)
The same micrograph (from EMPIAR 10335; Han et al. 2019) denoised using the pretrained model (left) or a model trained on this dataset (right). Training took eight minutes to complete 200 epochs with 100 training micrographs.