CryoSPARC Guide
Search…
⌃K

Troubleshooting

Overview of common issues and advice on how to resolve them.
Unless otherwise noted:
  • Log in to the workstation or remote node where cryosparc_master is installed.
  • Use the same non-root UNIX user account that runs the CryoSPARC process and was used to install CryoSPARC.
  • Run all commands on this page in a terminal running bash
In v4.0+, you can download error reporting information from within the application. For more details, see: Guide: Download Error Reports​

Common Issues

Cannot download or install CryoSPARC

Problems with the installation steps are indicated with the some of the following error messages:
"Couldn't connect to host" "Could not resolve host" {"success": false} "tar: This does not look like a tar archive" "Version mismatch! Worker and master versions are not the same. Please update." "An unexpected error has occurred."
Steps
  1. 1.
    If you have Conda installed, deactivate any active environments.
  2. 2.
    Check that your LICENSE_ID environment variable is set correctly with this command
    echo $LICENSE_ID
    Ensure the output exactly matches the CryoSPARC License ID issued to you over email.
  3. 3.
    Check your machine's connection to CryoSPARC's license servers at get.cryosparc.com with this curl command:
    curl https://get.cryosparc.com/checklicenseexists/$LICENSE_ID
    You should see the message {"success": true}. If instead you see {"success": false}, your license is not valid, so please check it has been entered correctly. If you see an error message like "Couldn't connect to host" or "Could not resolve host" check your Internet connection, firewall or ensure your IT department has the get.cryosparc.com license server domain whitelisted.

Cannot update CryoSPARC

Steps

CryoSPARC does not start or encounters error on startup

This can happen following a fresh install or recent update.
Steps
  1. 1.
    In a command line, run cryosparcm status
  2. 2.
    Check that the output looks like this
    ----------------------------------------------------------------------------
    CryoSPARC System master node installed at
    /home/cryosparcuser/cryosparc/cryosparc_master
    Current CryoSPARC version: v4.0.0
    ----------------------------------------------------------------------------
    ​
    CryoSPARC process status:
    ​
    app RUNNING pid 1223898, uptime 0:51:41
    app_api RUNNING pid 1224512, uptime 0:51:39
    app_api_dev STOPPED Not started
    app_legacy STOPPED Not started
    app_legacy_dev STOPPED Not started
    command_core RUNNING pid 1218914, uptime 0:51:56
    command_rtp RUNNING pid 1221639, uptime 0:51:48
    command_vis RUNNING pid 1220983, uptime 0:51:49
    database RUNNING pid 1217182, uptime 0:52:00
    ​
    ----------------------------------------------------------------------------
    License is valid
    ----------------------------------------------------------------------------
    ​
    global config variables:
    ​
    export CRYOSPARC_LICENSE_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    export CRYOSPARC_MASTER_HOSTNAME="localhost"
    export CRYOSPARC_DB_PATH="/home/cryosparcuser/cryosparc/cryosparc_database"
    export CRYOSPARC_BASE_PORT=39000
    export CRYOSPARC_INSECURE=false
    export CRYOSPARC_CLICK_WRAP=true
  3. 3.
    Check that all items under "CryoSPARC process status" that do not end in _dev or _legacy are RUNNING. If any are not, run cryosparcm restart
  4. 4.
    If any non-_dev /non-_legacy components have a status other than RUNNING (such as STOPPED or EXITED), check their log for errors. For example, this command checks for errors on the database process:
    cryosparcm log database
    (Press control C, then q to stop logging)
  5. 5.
    If the web interface is inaccessible, check firewall settings to ensure CryoSPARC's base port number (default 39000) is exposed for network access
Any error messages here could indicate specific configuration issues and may require re-installing CryoSPARC.
If at any point you see No command 'cryosparcm' found or command not found: cryosparcm:
  1. 1.
    Check that you are on the master node or workstation where cryosparc_master is installed
  2. 2.
    Run echo $PATH and check that it contains <installation directory>/cryosparc_master/bin
    $ echo $PATH
    /home/cryosparcuser/cryosparc/cryosparc_master/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  3. 3.
    ​Reinstall CryoSPARC if the above did not restore the the cryosparcm command

'User not found' error when attempting to log in

This error message occurs if the email address field does not match any existing users in your CryoSPARC instance. Use the CryoSPARC command-line to verify the details of your user account and change the email address or password if needed.
  1. 1.
    Run the following command in your terminal: cryosparcm listusers
  2. 2.
    If an email address is incorrect (e.g., mispelled or with an extra space at the beginning or end), modify it in the database. Run the following commands:
    • Log into the MongoDB shell: cryosparcm mongo
    • Once in the MongoDB shell, enter the following (replace the incorrect/correct email):
    db.users.update(
    { 'emails.0.address': '[email protected]' },
    { $set: { 'emails.0.address': '[email protected]' }
    })
    • Exit the MongoDB shell with exit
  3. 3.
    If you don't remember your password, reset it with the following command (replace with your email address and new password):
    cryosparcm resetpassword --email "<email address>" --password "<new password>"

License error or license not found

Follow the steps in this section when you see error messages that look like this:
"License is invalid." "License signature invalid." "Could not find local license file. Please re-establish your connection to the license servers." "Local license file is expired. Please re-establish your connection to the license servers." "Token is invalid. Another CryoSPARC instance is running with the same license ID."
Steps
Run cryosparcm licensestatus. This should result in "License is valid". If you see this error:
ServerError: Authentication failed
Your license ID is not entered configured correctly. Check the CRYOSPARC_LICENSE_ID entry in cryosparc_master/config.sh .
If you see this error:
WARNING: Could NOT verify active license
This includes a list of checks, the last of which will indicate a failure. Depending on which check failed, do one of the following:
  • Ensure you entered the license correctly during the installation step.
  • Check your Internet connection
  • Check your machine's connection to CryoSPARC's license servers at get.cryosparc.com with this curl command (substitute <license> with your unique license ID):
curl https://get.cryosparc.com/checklicenseexists/<license>
Look for the message message {"success": true}
If instead you see {"success": false}, your license is not valid so please check it has been entered correctly.
If you see an error message like "Couldn't connect to host" or "Could not resolve host" check your Internet connection, firewall or ensure your IT department has the get.cryosparc.com license server domain whitelisted.
If you see a license ID conflict such as
"Another cryoSPARC instance is running with the same license ID."
Check that no zombie processes are running on all nodes. First stop the running CryoSPARC instance:
cryosparcm stop
For every machine running CryoSPARC (including workers), list all processes associated with CryoSPARC with the following command:
ps -ax | grep cryosparc
If you see any processes that appear associated with CryoSPARC, kill them with the command kill <PID>, where <PID> is the process ID number at the beginning of each line.
For example, to kill the first python process in this process list
PID TTY STAT TIME COMMAND
12495 ? Sl 76:06 python -c import cryosparc_command.command_core as serv; serv.start(port=39002)
15879 ? Sl 379:49 mongod --dbpath /home/cryosparcuser/cryosparc/cryosparc_database --port 39001 --replSet meteor
17832 ? S 0:02 python -c import cryosparc_command.command_proxy as serv; serv.start(port=39004)
18361 ? Sl 288:40 python -c import cryosparc_command.command_rtp as serv; serv.start(port=39005)
19585 ? Sl 46:34 python -c import cryosparc_command.command_vis as serv; serv.start(port=39003)
Run
kill 12495
Alternatively, if you have the killall program installed, kill all running python processes with one of these commands:
killall -u $(whoami) python
# or
sudo killall -u <cryosparcuser> python
If the current Linux user runs CryoSPARC, use the first command. Otherwise use the second command, substituting <cryosparcuser> with the username that runs CryoSPARC.
Then start CryoSPARC on the master note or workstation:
cryosparcm start

Cannot queue or run job

Follow these steps when the CryoSPARC web interface is up and running normally and jobs may be created but do not run. These error messages may indicate this issue:
"list index out of range" "Could not resolve hostname ... Name or service not known"
A job that never changes from Queued or Launched status may also indicate this.
Steps
  1. 1.
    Ensure at least one worker is connected to the master. See the Installation page for details. Visit Manage > Resources to see what lanes are available
  2. 2.
    Check that all non-development CryoSPARC processes are running with the cryosparcm status command
  3. 3.
    (For master/worker setups) check that SSH is configured between the master and worker machines.
  4. 4.
    Check the log for the command_core log to find any application error messages
    cryosparcm log command_core
    (press Control + C on the keyboard followed by q to exit when finished)
  5. 5.
    If applicable, check that the cluster submission script is correct
  6. 6.
    Refresh job types and reload CryoSPARC:
    cryosparcm cli "refresh_job_types()"
    cryosparcm cli "reload()"
  7. 7.
    Force-reinstall master and worker dependencies. This can help when a worker was not correctly installed. On the workstation or master node run:
    cryosparcm forcedeps
    On worker nodes run:
    cryosparcw forcedeps
  8. 8.
    Restart CryoSPARC:
    cryosparcm restart
  9. 9.
    Clear the job and re-run it.

Job stuck in launched status

This indicates that CryoSPARC started the job process but the job encountered an internal error immediately after.

Steps

Check the job's Event Log for errors. If there are none, check the standard out log either from the interface under Job > Metadata > Job Log
​
or from the command line with
cryosparcm joblog PX JY
Substituting PX and JY for the project and job IDs, respectively.
Typically the errors here occur when the worker process cannot connect back to the master. Ensure there is a stable network connection between all machines involved. Ensure CryoSPARC was installed correctly and re-install if necessary.

Job runs but ends unexpectedly with status "Failed"

When a job fails, its job card in the interface turns red and the bottom of the job log includes an error message with the text Traceback (most recent call last)
Common failure reasons include:
  • Invalid or unspecified input slots
  • Invalid or unspecified required parameters, including file/folder paths
  • Incorrectly set up GPU (e.g., running a job on a node without enough GPUs or CUDA drivers not installed)
  • Another process taking up memory on a GPU
  • Cache not set up correctly for a worker
  • Lost connection to cryosparc_master
Example of failed jobs in a CryoSPARC workspace
Example of an error log entry at the bottom of a Failed job
Common job failure error messages:
"AssertionError: Child process with PID ... has terminated unexpectedly!" Job is unresponsive - no heartbeat received in 30 seconds.
Common error messages that indicate incorrectly configured GPU drivers:
"cuInit failed: unknown error" "no CUDA-capable device is detected" "cuMemHostAlloc failed: OS call failed or operation not supported on this OS" "cuCtxCreate failed: invalid device ordinal kernel.cu ... error: identifier "__shfl_down_sync" is undefined
Common error messages that indicate not enough GPU memory:
"cuMemAlloc failed: out of memory" "cuArrayCreate failed: out of memory" "cufftAllocFailed"
Steps
  1. 1.
    Ensure a CUDA version ≥10 and ≤11.7.1 is installed and running on the workstation or each worker node
  2. 2.
    Check the GPU configuration on the workstation or node where the job runs on. Log into that machine and navigate to the CryoSPARC installation directory. Run the cryosparcw gpulist command:
    cd /path/to/cryosparc_worker
    bin/cryosparcw gpulist
  3. 3.
    Run nvidia-smi to check the GPU and CUDA driver status.
    • Check that CUDA version is within the allowed range. CryoSPARC supports CUDA versions ≥10 and ≤11.7.1
    • Check that no other processes are using GPU memory. CryoSPARC-related process appear with process name "python"
    Example output of the nvidia-smi command, showing CUDA 10.2 and a CryoSPARC python process using ~2GB on GPU 0
    ​
    If you don't recognize the processes using GPU memory, run kill <PID>, substituting <PID> with the value under the Processes PID column
  4. 4.
    Check the Event log: Select the Job card in the CryoSPARC interface and press the Spacebar on your keyboard to see the log. Scroll down to the bottom and look for the failure reasons in red
    ​
  5. 5.
    Clear the job and select the "Build" status badge on the job card to enter the Job Builder
  6. 6.
    If the job failed with a GPU-related error and multiple GPUs are available, try running the job on a different GPU. Press Queue, switch to the "Run on specific GPU" Queue type and select one or more GPUs
  7. 7.
    If the job failed with AssertionError: Non-optional inputs from the following input groups and their slots are not connected then clear the job, enter the Builder and expand any input groups connected to the job. Missing required slots appear with text "Empty" and "Required"
  8. 8.
    Check the job parameters: To learn about setting specific parameters, hover over or touch the parameter names in the Job Builder to see a description of what they do
    On-hover description of the "Negative Stain Data" parameter for the "Import Movies" job
  9. 9.
    Find the target job type in this guide's Job Reference for more detailed descriptions of expected input slots and parameters: All Job Types in CryoSPARC.
  10. 10.
    Reduce the box-size of extracted particles. Some jobs need to fit thousands of particles in GPU memory at a time, and larger box sizes exceed GPU memory limits. Either extract with a smaller box size or with the Downsample Particles job.
  11. 11.
    Refresh job types and reload CryoSPARC:
    cryosparcm cli "refresh_job_types()"
    cryosparcm cli "reload()"
  12. 12.
    Look for extended error information with the cryosparcm joblog command (press Ctrl + C on the keyboard to exit when finished)
  13. 13.
    Check the network connection from the worker machine to the master
  14. 14.
    On occasion, a job fails due to an error in the CryoSPARC code (bug). The CryoSPARC team regularly releases updates and patches with bug fixes. Check for and install the latest update or patch. If you find a new bug, see the Additional Help section for advice.

Job stuck or taking a very long time

Due to their large sizes, cryo-EM datasets can take a long time to process with sub-optimal hardware or parameters. Here are some facilities that CryoSPARC provides for increasing speed/performance.
  • ​Connect workers with SSD cache enabled. This speeds up processing for extracted particles during 2D Classification, ab-initio reconstruction, refinement and more. Ensure the "Cache particle images on SSD" parameter is enabled under "Compute settings" for the target particle-processing job
  • Some jobs (motion correction, ctf estimation, 2D classification) can run on multiple GPUs. If your hardware supports it, increase the number of GPUs to parallelize over
2D Classification jobs support particle SSD caching and parallelizing over multiple GPUs.
  • Extracted particles with large box sizes (relative to their pixel size) take a long time to process. Consider Fourier-cropping (or "binning") the extracted particle blobs with the Downsample Particles job
  • Minimize the number of processes using system resources on the workstation or worker nodes
  • Check for zombie processes on worker machines. The process is similar to the steps under "Another CryoSPARC instance is running with the same license ID" under the License error or license not found section
  • Cancel the job, clear and re-queue

GPU Issues

cudaErrorInsufficientDriver

File "/u/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1711, in run_with_except_hook
run_old(*args, **kw)
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 129, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 997, in cryosparc_compute.engine.engine.process.work
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 106, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File "cryosparc_worker/cryosparc_compute/engine/gfourier.py", line 33, in cryosparc_compute.engine.gfourier.fft2_on_gpu_inplace
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/fft.py", line 102, in __init__
capability = misc.get_compute_capability(misc.get_current_device())
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/misc.py", line 254, in get_current_device
return drv.Device(cuda.cudaGetDevice())
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cudart.py", line 767, in cudaGetDevice
cudaCheckStatus(status)
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cudart.py", line 565, in cudaCheckStatus
raise e
skcuda.cudart.cudaErrorInsufficientDriver
This error usually indicates the NVIDIA driver you're using isn't compatible with the GPU and CUDA Toolkit version installed. This issue can be fixed by installing the latest NVIDIA Driver from the driver download page. ​
Once installed, you can recompile pyCUDA by using the newcuda command: cryosparcw reference.

SSD Cache Issues

In rare circumstances, jobs with "Cache particle images on SSD" enabled will not complete with one of the following scenarios:
  • The job logs "SSD cache : cache successfully synced" and never proceeds further
  • The job logs "SSD cache : requested files are locked for past Xs" and never proceeds further, even when there are no other jobs accessing the SSD cache
  • The job fails with an HTTP timeout error during the SSD caching step
These typically occur in long-active CryoSPARC instances with many projects and files on the SSD. There are several strategies to reduce these occurrences.

Option 1: Check SSD health

SSD's naturally degrade over time, the likelihood of failure increasing with heavy usage. Use a tool such as smartclt to check the SSD. If enough errors have accumulated, the SSD may have to be replaced.
CryoSPARC automatically removes files from the SSD that have not been accessed in a while (more than 30 days by default) each time the SSD cache system runs. If the SSD is very heavily used in particle processing jobs or by other external tools, leaving more free space available may extend its lifetime. This is possible with one or both of these strategies:
  • ​Reconnect the worker with the --ssdreserve flag set (default 10GB or 10000MB) to ensure CryoSPARC always leaves the given amount of free space on the SSD (will clean out old files to stay above the threshold)
  • Set the CRYOSPARC_SSD_CACHE_LIFETIME_DAYS environment variable in cryosparc_master/config.sh to clean up unused files on the SSD more frequently. The default value is 30 days

Option 2: Fix database inconsistencies

cryosparc_master uses its MongoDB database to coordinate SSD caching between multiple workers running in parallel. If a job fails unexpectedly during the SSD caching step, this could lead to database inconsistencies which prevent other jobs from proceeding.
To address these, try the following steps:
  1. 1.
    Ensure no jobs are running in CryoSPARC
  2. 2.
    In a terminal, enter cryosparcm mongo to enter the interactive database prompt
  3. 3.
    Enter the following command to check how many records are in an inconsistent state:
    db.cache_files.find({status: {$nin: ['hit', 'miss']}}).length()
  4. 4.
    If the result is not 0 (zero), enter the following command to fix them
    db.cache_files.updateMany({status: {$nin: ['hit', 'miss']}}, {$set: {status: 'miss'}})
  5. 5.
    Exit from the database prompt with Ctrl + D
  6. 6.
    Try re-running the problematic jobs

Option 3: Fully reset SSD cache system

If the previous options have no effect, fully reset the caching system with the following steps
  1. 1.
    Ensure no jobs are running in CryoSPARC
  2. 2.
    For each connected worker machine:
    • Navigate to the SSD cache directory containing CryoSPARC's cache files (e.g., /scratch/). This path was configured during installation time
    • Look for a directory named instance_<master hostname>:<master port + 1> e.g., instance_localhost:39001
    • Delete this directory and all its contents
  3. 3.
    In a terminal, enter cryosparcm mongo to enter the interactive database prompt
  4. 4.
    Enter the following command to clear out the cache records
    db.cache_files.deleteMany({})
  5. 5.
    Exit from the database prompt with Ctrl + D
  6. 6.
    Try re-running the problematic jobs

User Interface Error Logging

If you encounter a problem with the user interface, e.g., one or more elements of a page are not loading, etc., you can use the following steps to obtain debugging information.
  1. 1.
    Open the browser console
    • In Chrome, Firefox, and Safari this can be done by right clicking the page to open the browser context menu and selecting the Inspect option (Inspect Element in Safari) .
    • This will open up a “DevTools” panel used for inspecting and debugging in the browser. This panel includes a number of tabs at the top used to display different views. When opened using the context menu the current view will be the Elements tab. Click on the Console tab directly beside the Elements tab in order to view the web console. This is where errors, warnings, and general information about the page’s javascript code can be observed.
    • In order to keep the console clean in production we disable our development logs. Enable these logs by pasting the command
      window.localStorage.setItem('cryosparc_debug', true); into the browser console and then pressing the enter key on your keyboard. undefined will be logged below this command if it was submitted correctly.
    • Now reload the page and all of the development console logs and errors will be visible.
  2. 2.
    Save Console Output
    Please save console output as a .log file, including the type of browser (Chrome, Firefox, Safari, etc.) in the file name. The filename should be formatted as such: console_{browser}.log , eg. console_chrome.log .
    • Chrome: Right click anywhere in the console panel to open the context menu and select the Save As... option. This will allow you to save the entire output as a .log file.
    • Firefox: Right click on a console message to open the context menu and select the Save all Messages to File option. This will allow you to save the entire output as a .txt file as default (or .log file optionally).
    • Safari: Click and drag the cursor over all items in the console output so that the items are all highlighted blue. You can then right click on any of the highlighted items to open the context menu and select the Save Selected option to to save the entire output as a .txt file as default (or .log file optionally).
  3. 3.
    Save Network Output
    Navigate to the Network tab in the DevTools by selecting it from the tabs in the top bar of the panel. If the Network tab is not shown then it is likely hidden in the overflow menu (this appears when there is not enough space to display all of the tab options in the DevTools). Click the overflow menu button represented by two right chevrons (>>) and select the Network option.
    Please include the type of web browser (Chrome, Firefox, or Safari) in the name of the .har file you are saving. The filename should be formatted as such: network_{browser}.log , eg. network_chrome.log .
    • Chrome: Right click on any of the items in the network request table and then select Save all as HAR with context from the context menu.
      Make sure that the red record button at the top left of the panel has been activated. It will appear as a grey circle if it has not been activated, and a red circle if it has. The Preserve log checkbox must also be selected. Now reload the page and the network panel will be populated with all requests made and received by the browser
    • Firefox: Right click on any of the items in the network request table and then select Save All As HAR from the context menu.
    • Safari: Right click on any of the items in the network request table and then select Export HAR from the context menu.

Additional Help

For topics not covered above, get additional help through the CryoSPARC Discussion Forum:
If no related discussions exist, please create a new post. Review our Troubleshooting Guidelines for items to include in your post:
Previous
Guide: EMDB-friendly XML file for FSC plots
Next - Application Guide (v4.0+)
A Tour of the CryoSPARC Interface
Last modified 28d ago