Managing a CryoSPARC Live Session from the CLI (v5.0+)

This guide details how to automate the creation, configuration and overall management of a CryoSPARC Live session in Python via the CryoSPARC API.

Overview

This guide details how to automate the creation, configuration and overall management of a CryoSPARC Live session in Python via the CryoSPARC API. The instructions on this page are correct for CryoSPARC v5.0+.

Setup

Management of CryoSPARC Live uses the same CryoSPARC command line mechanics described in the cryosparcm cli reference. That page also describes three ways to access the CryoSPARC command line interface: i) cryosparcm cli "COMMAND", ii) cryosparcm icli for an interactive CryoSPARC shell, or iii) using cryosparc-tools in a Python script.

In addition, you can write a script that imports the necessary Python API client directly from CryoSPARC. To do so, first use the following code at the start of your script, for example in a file called automate_cryosparc_live.py:

from client.api_client import APIClient
from core import core

core.startup()

db = core.db
gfs = core.gridfs
api = APIClient(
    core.settings.api_base_url,
    auth=core.settings.api_admin_auth,
    headers={"license-id": core.settings.license_id},
)
# ...

Then, use CryoSPARC’s provided python environment to execute your script:

# Enter these in the command line
eval $(cryosparcm env)
export PYTHONPATH="${CRYOSPARC_ROOT_DIR}"
python automate_cryosparc_live.py

The remainder of this guide will assume that you are writing code within a script like the above where the api object is defined, or else using the interactive python shell provided by CryoSPARC (i.e., cryosparcm icli) where the api object is predefined.

Automate a CryoSPARC Live Session

Create A Project [Optional]

You will need a project to create your Live session. If you already have a project, you can skip this step.

Note that CryoSPARC will create a folder for the project inside the project_container_dir path you specify named after the UID of the project created (e.g., P12).

The definition of the project_description variable and the corresponding desc parameter are optional.

project_container_dir='/data/cryoSPARC_projects/'
project_title='EMPIAR-10025'
project_description='T20s Processing Project'

project = api.projects.create(project_container_dir, project_title, project_description)

Create a CryoSPARC Live Session

Note that the title and desc arguments are optional. project_uid can be defined explicitly for pre-existing projects, or by using the value returned by cli.create_empty_project(<..>)(see above).

project_uid='P12'# set this variable if you didn't already create a project in the above step
session_title='EMPIAR-10025 Live Processing Session'
session_description='Simulated live session'

session = api.sessions.create(project_uid, title=session_title, description=session_description)

Configure your CryoSPARC Live Session

1. Set the session’s compute configuration

phase_one_lane = 'gpu_lane_name'
phase_one_gpus = 2 # optional
phase_two_lane = 'gpu_lane_name'
phase_two_ssd = True # optional
auxiliary_lane = 'gpu_lane_name'
auxiliary_ssd = True # optional

live_compute_resources = {
	"phase_one_lane": 'gpu_lane_name',
	'phase_one_gpus': 2, # optional
	'phase_two_lane' : 'gpu_lane_name',
	'phase_two_ssd': True, # optional
	'auxiliary_lane' : 'gpu_lane_name',
	'auxiliary_ssd' : True # optional
}

session = api.sessions.update_compute_configuration(session.project_uid, session.session_uid, live_compute_resources)

1. Set the session’s exposure group parameters

   Note that the following lines change the exposure group parameters for the first exposure group (exp_group_id=1).

exposure_group_update = {
	'file_engine_watch_path_abs' : '/data/EMPIAR/10025/data/14sep05c_raw_196/',
	'file_engine_filter' : '*.frames.mrc',
	'gainref_path' : '/data/EMPIAR/10025/data/14sep05c_raw_196/norm-amibox05-0.mrc'
	
}
session = api.sessions.update_exposure_group(session.project_uid, session.session_uid, 1, exposure_group_update)

Set the session’s processing parameters

Note that the following parameters are required to be set before starting a session.

To see the full list of parameters available to be set, see below.

updated_params = {
	"psize_A": 0.6575,
	"accel_kv": 300,
	"cs_mm": 2.7,
	"total_dose_e_per_A2": 53,
	"diameter": 100, 
	"diameter_max": 200, 
	"box_size_pix": 440
}
session = api.sessions.update_session_params(session.project_uid, session.session_uid, updated_params)

Start the CryoSPARC Live Session

1. Start the session

   api.sessions.start(session.project_uid, session.session_uid)

   If you’ve configured your session properly, CryoSPARC Live will start to find movies and process data. At this point, you can navigate to the user interface and watch as exposures are processed. The next step would be to start Streaming 2D Classification and Streaming Homogeneous Refinement. You can also use the user interface to manually pick particles and use the template picker to identify particle locations.

2. Start Streaming 2D Classification

   class2d_job = api.sessions.setup_phase2_class2D(session.project_uid, session.session_uid)
   class2d_job = api.sessions.enqueue_phase2_class2D(session.project_uid, session.session_uid)

3. Start Ab-Initio Reconstruction

   1. Create a new initial volume

      abinit_job = api.sessions.setup_phase2_abinit(session.project_uid, session.session_uid)
      abinit_job = api.sessions.update_phase2_abinit_params(session.project_uid, session.session_uid, {"abinit_K": 3})
      abinit_job = api.sessions.enqueue_phase2_abinit(session.project_uid, session.session_uid, 

   2. Use an existing job’s volume output as the initial volume

      abinit_job = api.session.set_phase2_abinit_job(session.project_uid, session.session_uid, job_uid=abinit_job.uid)

4. Start Streaming Homogeneous Refinement

   Note that an initial volume must be set for the session before starting Streaming Homogeneous Refinement.

   refine_job = api.sessions.setup_phase2_refine(session.project_uid, session.session_uid)
   refine_job = api.sessions.update_phase2_refine_params(session.project_uid, session.session_uid, {"refine_symmetry": "D7"})
   refine_job = api.sessions.enqueue_phase2_refine(session.project_uid, session.session_uid)

Stop and clear the CryoSPARC Live Session

1. Pause the entire CryoSPARC Live Session

   session = api.sessions.pause(session.project_uid, session.session_uid)

2. Clear the CryoSPARC Live Session (Note that this will delete the session progress and all results, rendering exposure and particle jobs unusable)

   session = api.session.clear_session(session.project_uid, session.session_uid)

Export Exposures and Particles from the CryoSPARC Live Session

It is possible to export exposures and particles from a CryoSPARC Live Session for use in other

1. Export all exposures from the session

   api.sessions.create_and_enqueue_export_exposures(session.project_uid, session.session_uid)

2. Export all particles from the session

   api.sessions.create_and_enqueue_export_particles(session.project_uid, session.session_uid)

Graceful Pause

New in v5.0+, CryoSPARC Live now has the ability to automatically pause a session once it is idle for a certain period of time.

It can be configured programmatically as follows:

api.sessions.configure_auto_pause(session.project_uid, session.session_uid, auto_pause="graceful", auto_pause_after_idle_minutes=5)

Delayed start run configuration

New in v5.0+, CryoSPARC Live now has the abitity to wait until exposures are available on the filesystem before starting the session. This prevents the session from using hardware resources or license tokens until raw movie files are available for processing. It can be configured as follows:

api.sessions.set_session_phase_one_wait_for_exposures(session.project_uid, session.session_uid, True)

Parameters

The full list of session parameters that can be modified can be obtained with:

api.sessions.get_session_base_params()