Downloading and Installing cryoSPARC

Prepare for Installation
Log into the workstation where you would like to install and run cryoSPARC
1
ssh <cryosparcuser>@<cryosparc_server>
Copied!
<cryosparcuser> = cryosparcuser
The username of the account that cryoSPARC is to be installed by.
<cryosparc_server> = uoft
The hostname of the server where cryoSPARC will be installed on.
Determine where you'd like to install cryoSPARC.
1
cd <install_path>
Copied!
<install_path> = /home/cryosparcuser/cryosparc
This will be the root directory where all cryoSPARC code and dependencies will be installed.
Download And Extract The cryosparc_master And cryosparc_worker Packages Into Your Installation Directory
Export your License ID as an environment variable:
1
export LICENSE_ID="<license_id>"
Copied!
<license_id> = 682437fb-d6ae-47b8-870b-b530c587da94
This is the License ID issued to you, which you would have received in an email.
A unique license key is required for each individual crySPARC master instance.
Use curl to download the two files into tarball archives:
1
curl -L https://get.cryosparc.com/download/master-latest/$LICENSE_ID -o cryosparc_master.tar.gz
2
curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID -o cryosparc_worker.tar.gz
Copied!
Extract the downloaded files:
1
tar -xf cryosparc_master.tar.gz cryosparc_master
2
tar -xf cryosparc_worker.tar.gz cryosparc_worker
Copied!
Note: After extracting the worker package, you may see a second folder called cryosparc2_worker (note the 2) containing a single version file. This is here for backward compatibility when upgrading from older versions of cryoSPARC and is not applicable for new installations. 
You may safely delete the cryosparc2_worker directory.
Install The cryosparc_master Package
Follow these instructions to install the master package on the master node. If you are following the Single Workstation setup, once you run the installation command, cryoSPARC will be ready to use.
If you are installing cryoSPARC on a single workstation, you can use the Single Workstation instructions below that simplify installation to a single command. Otherwise, use the Master Node Only instructions here, and then continue with the further steps to install on worker nodes separately.
Single Workstation (Master and Worker combined)
Master Node Only
Single Workstation cryoSPARC Installation
1
cd cryosparc_master
2
​
3
./install.sh    --standalone \ 
4
                --license $LICENSE_ID \ 
5
                --worker_path <worker path> \ 
6
                --cudapath <cuda path> \ 
7
                --ssdpath <ssd path> \ 
8
                --initial_email <user email> \ 
9
                --initial_password <user password> \ 
10
                --initial_username "<login username>" \
11
                --initial_firstname "<given name>" \
12
                --initial_lastname "<surname>" \
13
                [--port <port_number>]
Copied!
Example command execution
1
./install.sh    --standalone \
2
                --license $LICENSE_ID \
3
                --worker_path /u/cryosparc_user/cryosparc/cryosparc_worker \
4
                --cudapath /usr/local/cuda \
5
                --ssdpath /scratch/cryosparc_cache \
6
                --initial_email "[email protected]" \
7
                --initial_password "Password123" \
8
                --initial_username "username" \
9
                --initial_firstname "FirstName" \
10
                --initial_lastname "LastName" \
11
                --port 61000
Copied!
Glossary/Reference
<worker_path> = /home/cryosparc_user/software/cryosparc/cryosparc_worker
the full path to the worker directory, which was downloaded and extracted in step c)
To get the full path, cd into the cryosparc_worker directory and run the command: pwd -P
<cuda_path> = /usr/local/cuda
path to the CUDA installation directory on the worker node (CUDA Requirements)
This path should not be the cuda/bin directory, but the cuda directory that contains both bin and lib64 subdirectories
<ssd_path> = /scratch/cryosparc_cache
path on the worker node to a writable directory residing on the local SSD (For more information on SSD usage in cryoSPARC, see X)
this is optional, and if omitted, specify the --nossd option to indicate that the worker node does not have an SSD
<initial_email> = [email protected]
login email address for first cryoSPARC webapp account
this will become an admin account in the user interface
<initial_username> = "FirstName LastName"
login username of the initial admin account to be created
ensure the name is quoted
<initial_firstname> = "LastName"
given name of the initial admin account to be created
ensure the name is quoted
<initial_lastname> = "LastName"
surname of the initial admin account to be created
ensure the name is quoted
<user_password> = Password123
temporary password that will be created for the <user_email> account
<port_number> = 39000
The base port number for this cryoSPARC instance. Do not install cryosparc master on the same machine multiple times with the same port number - this can cause database errors. 39000 is the default, which will be used if you don't specify this option.
Master Node cryoSPARC Installation
1
cd cryosparc_master
2
​
3
./install.sh --license $LICENSE_ID \
4
             --hostname <master_hostname> \
5
             --dbpath <db_path> \
6
             --port <port_number> \
7
             [--insecure] \
8
             [--allowroot] \
9
             [--yes] \
Copied!
Example command execution
1
./install.sh --license $LICENSE_ID \
2
             --hostname cryoem.cryosparcserver.edu \
3
             --dbpath /u/cryosparcuser/cryosparc/cryosparc_database \
4
             --port 45000 \
Copied!
Start cryoSPARC
1
./bin/cryosparcm start
Copied!
Create the first user
1
cryosparcm createuser --email "<user email>" \
2
                      --password "<user password>" \
3
                      --username "<login username>" \
4
                      --firstname "<given name>" \
5
                      --lastname "<surname>"
Copied!
Glossary/Reference
--license $LICENSE_ID
the LICENCE_ID variable exported in step a)
<master_hostname> = your.cryosparc.hostname.com
the hostname of the server where the master is to be installed on
<port_number> = 39000
The base port number for this cryoSPARC instance. Do not install cryosparc master on the same machine multiple times with the same port number - this can cause database errors. 39000 is the default, which will be used if you don't specify this option.
<db_path> = /u/cryosparcuser/cryosparc/cryosparc_database
the absolute path to a folder where the cryoSPARC database is to be installed. Ensure this location is in a readable and writeable location. The folder will be created if it doesn't already exist
<initial_email> = [email protected]
login email address for first cryoSPARC webapp account
this will become an admin account in the user interface
<initial_username> = "FirstName LastName"
login username of the initial admin account to be created
ensure the name is quoted
<initial_firstname> = "LastName"
given name of the initial admin account to be created
ensure the name is quoted
<initial_lastname> = "LastName"
surname of the initial admin account to be created
ensure the name is quoted
<user_password> = Password123
temporary password that will be created for the <user_email> account
--insecure
[optional] specify this option to ignore SSL certificate errors when connecting to HTTPS endpoints. This is useful if you are behind an enterprise network using SSL injection.
--allowroot
[optional] if you run the installation function with root privileges, it will fail. Specify this option to force cryoSPARC to be installed as the root user.
--yes
[optional] do not ask for any user input confirmations
[Optional] Re-load your bashrc. This will allow you to run the cryosparcm management script from anywhere in the system:
1
source ~/.bashrc
Copied!
[Optional] Set up recurring backups of the cryoSPARC database 
The cryoSPARC database holds metadata, images, plots and logs of jobs that have run. If the database becomes corrupt or lost due to to user error or filesystem issues, work that was done in projects can be lost. This can be recovered in two ways:
1.
In a new/fresh cryoSPARC instance, import the project directories of projects from the original instance. This should resurrect all jobs, outputs, inputs, metadata, etc. Please note: user and instance level settings and configuration will be lost.
2.
Maintain a backup (daily or weekly recommended) of the cryoSPARC database. You can use the backup functionality documented here as well as a cron job or other recurring scheduler to run the backup command.
Access the User Interface
After completing the above, navigate your browser to http://<workstation_hostname>:39000 to access the cryoSPARC user interface.
Accessing the cryoSPARC User Interface
If you were following the Single Workstation install above, your installation is now complete.
Install The cryosparc_worker Package
Log onto the worker node (or a cluster worker node if installing on a cluster) as the cryosparcuser, and run the installation command.
Installing cryosparc_worker requires a NVIDIA GPU and CUDA Toolkit version ≥10
GPU Worker Node cryoSPARC Installation
1
cd cryosparc_worker
2
​
3
./install.sh --license $LICENSE_ID \
4
             --cudapath <cuda_path> \
5
             [--yes]
Copied!
Worker Installation Glossary/Reference
--license $LICENSE_ID
The LICENCE_ID variable exported in step a)
<cuda_path> = /usr/local/cuda
Path to the CUDA installation directory on the worker node
Note: this path should not be the cuda/bin sub directory, but the CUDA directory that contains both bin and lib64 subdirectories
--yes
[optional] do not ask for any user input confirmations
Connecting A Worker Node
Connect A Managed Worker to CryoSPARC
Ensure cryoSPARC is running when connecting a worker node. Log into the worker node and run the connection function (substitute the placeholders with your own configuration parameters; details below):
1
cd cryosparc_worker
2
​
3
./bin/cryosparcw connect --worker <worker_hostname> \
4
                         --master <master_hostname> \
5
                         --port <port_num> \
6
                         --ssdpath <ssd_path> \
7
                         [--update] \
8
                         [--sshstr <custom_ssh_string> ] \
9
                         [--nogpu] \
10
                         [--gpus <0,1,2,3> ] \
11
                         [--nossd] \
12
                         [--ssdquota <ssd_quota_mb> ] \
13
                         [--ssdreserve <ssd_reserve_mb> ] \
14
                         [--lane <lane_name> ] \
15
                         [--newlane]
Copied!
Worker Connection Glossary/Reference
1
--worker <worker_hostname> = worker.cryosparc.hostname.com
2
# [required] hostname of the worker node to connect to your 
3
# cryoSPARC instance
4
​
5
--master <master_hostname> = your.cryosparc.hostname.com
6
# [required] hostname of the server where the master is 
7
# installed on
8
​
9
--port <port_number> = 39000
10
# [required] base port number for the cryoSPARC instance you 
11
# are connecting the worker node to
12
​
13
--ssdpath <ssd_path>
14
# [optional] path to directory on local SSD
15
# replace this with --nossd to connect a worker node without an SSD
16
​
17
--update
18
# [optional] use when updating and existing configuration
19
​
20
--sshstr <custom_ssh_string>
21
# [optional] custom SSH connection string such as [email protected] 
22
​
23
--nogpu
24
# [optional] connect worker with no GPUs 
25
​
26
--gpus 0,1,2,3
27
# [optional] enable specific GPU devices only 
28
​
29
# For advanced configuration, run the gpulist command:
30
# $ bin/cryosparcw gpulist
31
​
32
# Detected 4 CUDA devices.
33
​
34
#   id           pci-bus  name
35
#   ---------------------------------------------------------------
36
#       0      0000:42:00.0  Quadro GV100
37
#       1      0000:43:00.0  Quadro GV100
38
#       2      0000:0B:00.0  Quadro RTX 5000
39
#       3      0000:0A:00.0  GeForce GTX 1080 Ti
40
#   ---------------------------------------------------------------
41
# This will list the available GPUs on the worker node, and their 
42
# corresponding numbers. Use this list to decide which GPUs you wish 
43
# to enable using the --gpus flag, or leave this flag out to enable all GPUs.
44
​
45
--nossd
46
# [optional] connect a worker node with no SSD 
47
​
48
--ssdquota <ssd_quota_mb>
49
# [optional] quota of how much SSD space to use (MB) 
50
​
51
--ssdreserve <ssd_reserve_mb>
52
# [optional] minimum free space to leave on SSD (MB) 
53
​
54
--lane <lane_name>
55
# [optional] name of lane to add worker to 
56
​
57
--newlane
58
# [optional] force creation of a new lane if the lane specified by --lane 
59
# does not exist
Copied!
Update a Managed Worker Configuration
To update an existing managed worker configuration, use cryosparcw connect with the --update flag and the field you would like to update.
For example:
1
cd cryosparc_worker
2
./bin/cryosparcw connect --worker <worker_hostname> \
3
                         --master <master_hostname> \
4
                         --port <port_num> \
5
                         --update
6
                         --ssdquota 500000
Copied!
Connect a Cluster to CryoSPARC
Once the cryosparc_worker package is installed, the cluster must be registered with the master process. This requires a template for job submission commands and scripts that the master process will use to submit jobs to the cluster scheduler.
To register the cluster,  provide cryoSPARC with the following two files and call the cryosparcm cluster connect command: 
cluster_info.json
cluster_script.sh
The first file (cluster_info.json) contains template strings used to construct cluster commands (e.g., qsub, qstat, qdel etc., or their equivalents for your system). The second file (cluster_script.sh) contains a template string to construct appropriate cluster submission scripts for your system. The jinja2 template engine is used to generate cluster submission/monitoring commands as well as submission scripts for each job.
Create the files
The following fields are required to be defined as template strings in the configuration of a cluster. Examples for PBS are given; use any command required for your particular cluster scheduler. Note that parameters listed as "optional" can be omitted or included with their value as null.
See the following page for examples of these files:
cryoSPARC Cluster Integration Script Examples
cluster_info.json
1
name               :  "cluster1"
2
# string, required
3
# Unique name for the cluster to be connected (multiple clusters can be 
4
# connected)
5
​
6
worker_bin_path    :   "/path/to/cryosparc_worker/bin/cryosparcw"
7
# string, required
8
# Path on cluster nodes to the cryosparcw script
9
​
10
cache_path         :   "/path/to/local/SSD/on/cluster/nodes"
11
# string, optional
12
# Path on cluster nodes that is a writable location on local SSD on each 
13
# cluster node. This might be /scratch or similar. This path MUST be the 
14
# same on all cluster nodes. Note that the installer does not check that 
15
# this path exists, so make sure it does and is writable. If you plan to 
16
# use the cluster nodes without SSD, you can omit this field.
17
​
18
cache_reserve_mb   :   10000
19
# integer, optional
20
# The size (in MB) to initially reserve for the cache on the SSD. This 
21
# value is 10GB by default, which means cryoSPARC will always leave at
22
# least 10GB of space on the SSD free.
23
​
24
cache_quota_mb     :   1000000
25
# integer, optional
26
# The maximum size (in MB) to use for the cache on the SSD.
27
​
28
send_cmd_tpl       :   "ssh loginnode {{ command }}"
29
# string, required
30
# Used to send a command to be executed by a cluster node (in case the 
31
# cryosparc master is not able to directly use cluster commands). If your 
32
# cryosparc master node is able to directly use cluster commands 
33
# (like qsub etc) then this string can be just "{{ command }}"
34
​
35
qsub_cmd_tpl       :   "qsub {{ script_path_abs }}"
36
# string, required
37
# The command used to submit a job to the cluster, where the job 
38
# is defined in the cluster script located at {{ script_path_abs }}. This 
39
# string can also use any of the variables defined in cluster_script.sh 
40
# that are available when the job is scheduled (e.g., num_gpus, num_cpus, etc.,)
41
​
42
qstat_cmd_tpl      :   "qstat -as {{ cluster_job_id }}"
43
# string, required
44
# Cluster command that will report back the status of cluster job with its id 
45
# {{ cluster_job_id }}.
46
​
47
qdel_cmd_tpl       :   "qdel {{ cluster_job_id }}"
48
# string, required
49
# Cluter command that will kill and remove {{ cluster_job_id }} from the 
50
# queue.
51
​
52
qinfo_cmd_tpl      :   "qstat -q"
53
# string, required
54
# General cluster information command
55
​
56
transfer_cmd_tpl   :   "scp {{ src_path }} loginnode:{{ dest_path }}"
57
# string, optional
58
# Command that can be used to transfer a file {{ src_path }} on the cryosparc 
59
# master node to {{ dest_path }} on the cluster nodes. This is used when the 
60
# master node is remotely updating a cluster worker installation. This is 
61
# optional; if it is incorrect or omitted, you can manually update the 
62
# cluster worker installation.
Copied!
Along with the above commands, a complete cluster configuration requires a template cluster submission script. The script must send jobs into your cluster scheduler queue and mark them with the appropriate hardware requirements. The cryoSPARC internal scheduler submits jobs with this script as their inputs become ready. The following variables are available for use used within a cluster submission script template. When starting out, example templates may be generated with the commands explained below.
cluster_script.sh
1
{{ script_path_abs }}    # absolute path to the generated submission script
2
{{ run_cmd }}            # complete command-line string to run the job
3
{{ num_cpu }}            # number of CPUs needed
4
{{ num_gpu }}            # number of GPUs needed.
5
{{ ram_gb }}             # amount of RAM needed in GB
6
{{ job_dir_abs }}        # absolute path to the job directory
7
{{ project_dir_abs }}    # absolute path to the project dir
8
{{ job_log_path_abs }}   # absolute path to the log file for the job
9
{{ worker_bin_path }}    # absolute path to the cryosparc worker command
10
{{ run_args }}           # arguments to be passed to cryosparcw run
11
{{ project_uid }}        # uid of the project
12
{{ job_uid }}            # uid of the job
13
{{ job_creator }}        # name of the user that created the job (may contain spaces)
14
{{ cryosparc_username }} # cryosparc username of the user that created the job (usually an email)
Copied!
Note: The cryoSPARC scheduler does not assume control over GPU allocation when spawning jobs on a cluster. The number of GPUs required is provided as a template variable. Either your submission script or your cluster scheduler is responsible for assigning GPU device indices to each job spawned based on the provided variable. The cryoSPARC worker processes that use one or more GPUs on a cluster simply use device 0, then 1, then 2, etc.
Therefore, the simplest way to correctly allocate GPUs is to set sets the CUDA_VISIBLE_DEVICES environment variable in your cluster scheduler or submission script. Then device 0 is always the first GPU that a running job must use. The example script for pbs clusters (generated as below) shows how to check which GPUs are available at runtime and automatically select the next available device.
Load the scripts and register the integration
To create or set a configuration for a cluster in cryoSPARC, use the following commands. 
The command cryosparcm cluster connect attempts reading cluster_info.json and cluster_script.sh from the current working directory.
1
cryosparcm cluster example <cluster_type>
2
# dumps out config and script template files to current working directory
3
# examples are available for pbs and slurm schedulers but others should 
4
# be very similar
5
​
6
cryosparcm cluster dump <name>
7
# dumps out existing config and script to current working directory
8
​
9
cryosparcm cluster connect
10
# connects new or updates existing cluster configuration, 
11
# reading cluster_info.json and cluster_script.sh from the current directory, 
12
# using the name from cluster_info.json
13
​
14
cryosparcm cluster remove <name>
15
# removes a cluster configuration from the scheduler
Copied!
Update a Cluster Configuration
To update an existing cluster integration, call the cryosparcm cluster connect commands with the updated cluster_info.json and cluster_script.sh in the current working directory. 
Note that the name field from cluster_info.json must be the same in the cluster configuration to update
If you don't already have the cluster_info.json and cluster_script.sh files in your current working directory, you can get them by running the command cryosparcm cluster dump <name>