> For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc.md).

# Guide: SSD Particle Caching in CryoSPARC

Overview of how SSD particle caching works, how much SSD space you need, configuration options and troubleshooting.

## Why is particle caching effective?

For classification, refinement, and reconstruction jobs that use particles, local worker SSDs significantly speed up computation: Many cryo-EM algorithms rely on random-access patterns and multiple passes though the data, rather than sequentially reading the data once. 

When installing CryoSPARC, you may provide an SSD path. This should be a fast drive location available locally on the worker or workstation. Avoid using Network-Attached Storage. CryoSPARC manages this SSD cache on each worker node. When you run jobs with the `Cache particle images on SSD` option enabled, CryoSPARC copies and reads the job's input particles at the specified SSD path. If multiple jobs within the same project require the same particles, CryoSPARC re-uses cached files without copying. If more space is needed, CryoSPARC deletes previously cached data that is no longer in use.

{% hint style="info" %}
Setting up SSD cache is optional on a per-worker node basis, **but it is highly recommended,** especially in projects where multiple jobs using the same particles run on the same machine.

Only jobs that require picked and extracted particle images can use SSD cache. Worker nodes reserved for pre-processing (motion correction, CTF estimation, particle picking, etc.) do not need SSD cache. 

Caching can slow down processing if the SSD is not much faster than the project directory storage.
{% endhint %}

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FTcXyDye01nyPKl4a6hzx%2Fv5-0-0-ssd-cache-demo.avif?alt=media&amp;token=2bf014a5-45e9-45c0-b37a-5cf11123ded8" alt="Job event log with SSD cache status output."><figcaption><p>Screen recording of the SSD cache step in a job's event log.</p></figcaption></figure>

## Hardware

The size of your typical cryo-EM single particle datasets will inform the size of SSD you choose to use. To store the largest of particle stacks, we recommend 2TB capacity SSDs, or larger. You may calculate the exact size of a particle dataset with the following formula:

$$
Dataset\ Size =bpp*box\_size^2*num\_particles + header\_length\*num\_files
$$

Where&#x20;

* $$bpp$$ is "bytes per pixel", equal to 2 when extracting particles in float16 (default) and 4 for float32.
* $$box\_size$$ is the $$x$$ or $$y$$ dimension of the extracted particle image, specified when extracting.
* $$num\_particles$$ is the number of all particles in the stack, including particles marked as rejected but still present in the stacks. One may exclude rejected particles by restacking or re-extracting the set of accepted particles (see below).
* $$header\_length$$ is the size of an MRC file's header section, always 1024 bytes.
* $$num\_files$$ is the number of MRC files the stack is distributed between. Typically matches the number of exposures used to extract the particles, unless the particles have been restacked (see below).

For example: A 1,000,000 float16 particle dataset with box size 256 contained in 10,000 MRC files will have a total size of 131 GB

$$
2*256^2*1,000,000+1024\*10,000=131,082,240,000 \ bytes
$$

For example: A 2,000,000 float32 particle dataset with box size 432 contained in 10,000 MRC files will have a total size of 1.5 TB

$$
4*432^2*2,000,000+1024\*10,000=1,493,002,240,000 \ bytes
$$

## Configuration

### Installation

When [installing a CryoSPARC workstation](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc.md) or [connecting a worker](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc.md), use the `--ssdpath` option to specify the path of your SSD drive.

This allows queuing jobs to this workstation/connected worker with `Cache particle images to SSD` enabled.

* [Downloading and Installing CryoSPARC](/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc.md#single-workstation-cryosparc-installation)
* [cryosparcw reference (v5.0+)](/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcw-reference-v5.0.md#cryosparcw-connect)
* [cryosparcm reference (v5.0+)](/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcm-reference-v5.0.md#cryosparcm-worker-connect)

### Advanced Parameters

Specify the following options to fine-tune your SSD cache usage:

* `--ssdquota`: The maximum amount of space that CryoSPARC can use on the SSD (MB)
* `--ssdreserve`: The minimum amount of free space to leave on the SSD (MB)

Use these when installing CryoSPARC on a shared compute environment, where multiple applications access the SSD. The `--ssdreserve` option defaults to 10,000 MB; increasing this also prevents regularly leaving the SSD close to full, which may extend its lifetime.

### Updating Configuration

Update the SSD configuration at any time by re-running the `cryosparcw connect`  or `cryosparcm worker connect` command.

* [cryosparcw reference (v5.0+)](/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcw-reference-v5.0.md#cryosparcw-connect)
* [cryosparcm reference (v5.0+)](/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcm-reference-v5.0.md#cryosparcm-worker-connect)

Note that CryoSPARC v4.7 and older requires the `--update` flag to change configuration for an existing worker.

## Usage

### Use the SSD cache system when running a job

When building jobs that process particles stacks (for example: Ab-Initio, Homogeneous Refinement, 2D Classification, 3D Variability), look for the `Cache particle images on SSD` parameter under "Compute Settings". Disable this option to load particles from their original location instead.&#x20;

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FHMetwqHOJZx7q3gtioUv%2Fv5-0-0-ssd-cache-param.png?alt=media&amp;token=be3f0026-7220-42e4-a48a-8e044d2b2ea6" alt="Cache particle images on SSD parameter in the job builder." width="563"><figcaption></figcaption></figure>

### Set a default parameter for the project

By default, the `Cache particle images on SSD` parameter is on for every job that supports it. To turn this option off for all jobs in a project, set the "Enable SSD Caching" project-level default from the current project's sidebar.

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FhXft9PApH4GkSN1ZyyMH%2Fv5-0-0-project-params-default.png?alt=media&amp;token=2bad3ab7-4f0b-4c57-8124-cffc4327c4b3" alt="Project sidebar with project level parameter default options." width="375"><figcaption></figcaption></figure>

You may also enable Float16 outputs by default to reduce extracted particle sizes by half compared to the default Float32.

## Tips and Tricks

### Consolidating a Particle Stack

When particle stacks exceed the available space on the SSD, you may consolidate the particle stack with a [Restack Particles job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-restack-particles). This only works if the current particle stack is a subset of the original particle stack.

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2Fp3wUGEWWCQV8JIh4EsAu%2Fv4.1.2-restack-particles-1.png?alt=media&amp;token=e3eb4ed8-3e4a-4163-9527-421aac89ef40" alt=""><figcaption><p>Input particles can be distributed across many .mrc files, and often only a subset of each file is being used for downstream operations. Restack will consolidate the particles from many files into a smaller number of files that don’t include unused particles, therefore taking up less disk space.</p></figcaption></figure>

When the cache system starts running, you may see a failure with message `SSD cache needs additional X GB` , with `X` size much larger than you expect for the current stack. This may occur if, for example, you ran an "Select 2D Classes" job to only select a subset of extracted particles. The list of particles in the output is smaller, but Select 2D keeps the reference to the original extracted particle stacks. This is to avoid duplicating potentially large amounts of particle images in each job. However, the SSD system is designed to work on whole files: When using this particle subset in further processing, the full original particle stack must be cached.

To consolidate the particle stack, build a "Restack Particles" job, connect the particles subset, and run the job. Connect the restacked output to the caching job instead, then re-run.

### Save results in Float16

Another strategy to reduce SSD cache requirements is to extract particles in using 16-bit floating-point per-pixel precision (v.s. the default 32-bit precision). This reduces disk usage by half and is not expected to significantly impact reconstruction quality. 16-bit images also take half the time to copy the SSD. See [Tutorial: Float16 Support](/processing-data/tutorials-and-case-studies/tutorial-float16-support.md) for details.

To save data in float16 format, look for a toggle parameter called “Save results in 16-bit floating point” in motion correction or extraction jobs, and enable it.

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FfJv8XOaYeMjXdLBPg9qO%2Fv4-4-0-float16-builder-1.png?alt=media&amp;token=59fbfd86-bdd9-48b0-bcde-718e5fd96187" alt=""><figcaption></figcaption></figure>

You may also make this the project-level default ([see above](#set-a-default-parameter-for-the-project)).

### Dynamic SSD Cache Paths

On some systems it is not possible to know the SSD cache path ahead of time. Instead, a dynamically-generated path is available for jobs to use at run-time.

To prompt a job to use this path, make the path available via environment variable `CRYOSPARC_SSD_PATH`. You can add this variable directly in `cryosparc_worker/config.sh` :

```
# cryosparc_worker/config.sh
export CRYOSPARC_LICENSE_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export CRYOSPARC_USE_GPU=true
export CRYOSPARC_CUDA_PATH="/usr/local/cuda"
export CRYOSPARC_SSD_PATH="$CUSTOM_DYNAMIC_SSD_PATH"
```

You may also add it to the worker's global shell configuration.

{% hint style="info" %}
You must still provide the `--ssdpath` option during installation. Specify a dummy path like `/tmp`; this will not be used if `CRYOSPARC_SSD_PATH` is available&#x20;
{% endhint %}

### Increase or Reduce Cache Files Lifetime

In CryoSPARC v3.3+, jobs that use the SSD cache will automatically remove cache files that have not been accessed in over 30 days. If projects on your instance are actively worked on for more or less time, you may change the cache file lifetime by adding the following line to `cryosparc_master/config.sh`:

```
export CRYOSPARC_SSD_CACHE_LIFETIME_DAYS=15
```

Substitute `15` with the number of days your projects are typically worked on.

### Leveraging Multiple Threads to Copy Particles

In CryoSPARC v4.3+, multiple threads (default 2) are used to copy particles from the project directory to the local cache device. Increasing the number of threads is recommended for project directories hosted on specialized file systems designed for parallel reads.

To modify the number of threads used, add the following line to `cryosparc_worker/config.sh,` where `num_threads` is the number of threads (e.g., 12) to copy files in parallel:&#x20;

```
export CRYOSPARC_CACHE_NUM_THREADS=num_threads
```

Specify `export CRYOSPARC_CACHE_NUM_THREADS=1` to turn off multithreading and copy particle files sequentially.

## Troubleshooting

This section describes some messages or errors you may encounter while a job is preparing its SSD cache, what they mean and how to address them.

### `X of additional cache space required for Y file(s)`&#x20;

This temporary message means the SSD is currently in use by other jobs and there is not enough remaining space to copy the current job's files. Once the other jobs finish, the current job will remove their cached files so that it has enough space to proceed. If you regularly encounter this, consider running the jobs on other workers, increasing the SSD quota or installing a higher-capacity SSD.

### `X pending file(s) locked by other jobs`&#x20;

This temporary message means some required files are already being cached by another job. For example, if two Refinement jobs run at the same time on the same worker (Job A and Job B) using the same input particles, both jobs try acquire a lock on those particles. If Job A acquires the lock for the files first, it starts copying them and Job B shows this message. When Job A finishes copying the files, it unlocks them. Job B is unlocked and finds that the particles are already on the SSD, so it skips the copy step.

### `SSD cache needs additional X but drive can only be filled to Y`

This error message means the particle stack exceeds the size of the SSD or the configured quota/reserve. You may disable SSD cache for the job as noted, or restack the particles to reduce the cache requirements.

### `SSD cache needs additional X but drive has Y free. CryoSPARC can only free a maximum of Z. This may indicate that programs other than CryoSPARC are using the SSD`

This error message means that the requested files *do not* exceed SSD size or quota, but there are files outside the configured SSD cache directory that CryoSPARC cannot delete. These are likely generated by other applications or CryoSPARC instances which also access the SSD. It may be necessary to manually delete these files to free up SSD space.

See [Troubleshooting](/setup-configuration-and-management/troubleshooting.md#ssd-cache-issues) for full troubleshooting steps.

## FAQ

**Is it safe to manually delete cache files for completed or unqueued/cleared jobs?**&#x20;

Yes, it is safe to delete cache files while no jobs are running.

Source:

[How to clear the cache in v2?](https://discuss.cryosparc.com/t/how-to-clear-the-cache-in-v2/2161)